Selection of individuals from a pool of candidates in a competition system

ABSTRACT

An integrated highly automated competition system allows for selection of one or more competition winners. Documents can be submitted electronically or by hardcopy. Document submissions are checked against databases to determine duplicate existence by a variety of information extracted from the documents. Candidates are filtered by one or more filters requiring candidates meet completeness, qualification, eligibility, and partner/discretionary standards. Candidates are evaluated by one or more readers and scored on a plurality of cognitive and non-cognitive variables. Readers are monitored and corrective action is taken when reader evaluations are determined to require corrective action. Candidates are selected by a two stage process first using raw read scores and then using a successive weighted read score iteration or tie-breaking stage. Winning candidates are subject to progress tracking and re-qualification prior to receiving award payments.

This application is a Continuation of application Ser. No. 09/996,907,filed on Nov. 30, 2001, the entire contents of which are incorporated byreference.

FIELD OF THE INVENTION

The present invention relates to the evaluation of human resources, andmore particularly to the automation-enhanced selection of individualsfrom a pool of competition candidates based on evaluations of cognitiveand non-cognitive variables relative to a competition standard.

BACKGROUND OF THE INVENTION

The appropriate selection of individuals or candidates for employment,appointment to political positions, promotions, receiving awards orscholarships, and so on has traditionally been a difficult problem.

One trend facing competition systems is that of ever-increasing numbersof applications being received by competition systems. This results inproblems of data handling, of selecting the best candidates from anever-increasing number of highly-qualified applicants, and also ofeliminating an ever-increasing number of lower-quality, ineligible, orincompletely-responsive candidates.

The problem of candidate selection is compounded by candidate fraudwhich is becoming increasingly sophisticated. The forging of documents,the use of documents or credentials of others having similar names ordemographic information, and so on complicates competition systems dueto the detection and resolution requirements which must be imposed, aswell as the resultant unfairness when such fraud goes undetected.

Another problem is the occurrence of duplicate forms. This often occursas individuals apply multiple times in order to correct information orperhaps even in the belief that this can aid their chances of success.

The detection of both duplicates and fraud is further complicated,however, in that situations arise where two or more legitimatecandidates sometimes appear identical from their submitted papers. Suchsituations often arise in connection with twins having similar oridentical names, for example.

The selection of candidates for positions of future responsibility suchas political appointees, military or civilian officers or leaders,corporate management, academic scholars, and so on has at its root theproblem of selecting the best candidate based on predicted futureperformance. Studies have shown that non-traditional variables such asnon-cognitive variables are sometimes a better predictor of futureperformance than traditionally-used variables.

The use of these variables, as well as the use of traditional butsubjective variables, such as essay tests, has a problem because thenature of their evaluation is subjective and thus difficult, even for asingle evaluator, to do in a fair and consistent manner. This problemincreases with the number of evaluators and encompasses the problems ofevaluator selection, training, and monitoring. Studies have shown, forexample, that evaluators sometimes vary their decision methodologies intheir evaluations, especially just before breaks or at the end of theday. Furthermore, studies have also shown that evaluators may evaluatenon-cognitive variables more harshly for candidates of one ethnic orcultural background as compared with candidates of other ethnic orcultural backgrounds. The monitoring of evaluators is additionallycomplicated as system generally don't have the infrastructure necessaryto allow real-time monitoring in a non-invasive way or the ability toimplement corrective measures in real-time.

Additionally, traditional systems are lacking in methodologies to trackboth competition winners and competition losers for later use inproducing statistical support for and improvement of system selectioncriteria.

From the above, it is evident that improvements are needed in the areasof data management (i.e. duplicate identification, false-duplicateidentification, and fraud identification), candidate pool reduction,evaluator management, candidate selection, and winning-candidateprogress tracking.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a block diagram of an embodiment of a system for theselection of individuals from a pool of candidates;

FIG. 1 a shows a block diagram of an embodiment for database 104 of FIG.1;

FIG. 2 shows a block diagram of an embodiment for a high-level softwarearchitecture for the system of FIG. 1;

FIGS. 3 a-3 b show a block diagram in greater detail of the softwarearchitecture of FIG. 2;

FIGS. 3 c-3 d show the block diagram of FIGS. 3 a-3 b emphasizing theinterconnections of the reporting module (280).

FIG. 4 shows a general flowchart of steps traversed by an exemplarymethod in operation of the systems of FIGS. 1-2;

FIG. 5 shows a flowchart of steps traversed by an exemplary method inthe operation of electronic reception of documents in the systems ofFIGS. 1-2;

FIG. 6 shows a flowchart of steps traversed by an exemplary method inthe operation of hardcopy reception in the systems of FIGS. 1-2;

FIG. 7 shows a flowchart of the steps traversed in an exemplaryalgorithm for the storing and duplicate detection for the submissionprocedures shown in FIGS. 5-6;

FIG. 7 a shows a flowchart of an exemplary embodiment for possibleduplicate document review;

FIG. 8 shows a flowchart of steps traversed in an exemplary algorithmthe grouping and level 1 filtering for candidate package completenessshown in FIG. 4;

FIG. 9 shows a flowchart of steps traversed in an exemplary algorithm inthe operation of the level 2-3 filtering of electronic documents shownin FIG. 4;

FIG. 10 shows a flowchart of steps traversed in an exemplary method inoperation of the reading process of FIG. 4;

FIG. 11 shows a flowchart of steps traversed in an exemplary method inoperation of the selection and training of readers to participate in theprocess of FIG. 8;

FIGS. 12 a-12 b show a flowchart of steps traversed in an exemplaryalgorithm in operation of finalist selection of FIG. 4;

FIGS. 12 c-12 e shows an example of the steps traversed in anapplication of the tie-breaking procedure of FIG. 12 b.

FIG. 13 shows a flowchart of steps traversed in an exemplary method inoperation of competition winner confirmation of FIG. 4;

FIG. 14 shows a flowchart of steps traversed in an exemplary method inoperation of competition winner tracking of FIG. 4;

FIG. 14 a shows a flowchart of the steps traversed in an exemplarymethod for the tracking of competition winners and non-selectedapplicants;

FIG. 15 shows an exemplary display presented to an evaluator;

FIGS. 16-17 show exemplary displays for use during determination ofevaluator eligibility;

FIGS. 18-19 show exemplary displays for use during determination ofcandidate eligibility;

FIGS. 20-21 show exemplary displays for use during evaluation ofcandidate packages;

FIGS. 22-25 show exemplary displays of candidate progress detailreports;

FIGS. 26-28 show exemplary displays of completed applicant packagereports;

FIGS. 29-30 show exemplary displays of possible duplicate documentreports;

FIGS. 31-34 show exemplary display of candidate detail listing reports;

FIG. 35 shows an exemplary display of a nominator floater with possiblekey form report;

FIGS. 36-37 show exemplary display of nominator floater detail reports;

FIGS. 38-40 show exemplary display of nominator detail reports;

FIG. 41 shows an exemplary display of a recommender floater withpossible key form report;

FIGS. 42-43 show exemplary display of recommender floater detailreports; and

FIGS. 44-46 show exemplary display of recommender detail reports.

DETAILED DESCRIPTION

Shown and described are embodiments, automated and partially-automated,of a system and method for selecting one or more candidates from a poolof candidates. It is noted that while this disclosure illustrates thepresent invention by examples drawn mainly from academic scholarshipcompetitions, the present invention has application in many fieldsoutside of academics such as, but not limited to: employee hiring;employee to job, position, or task matching; military officer selection;political appointee selection; personal date selection; personal friendselection; and so forth.

Referring to FIG. 1, shown is a block diagram of an embodiment for acompetition system (100) for the selection of individuals from a pool ofcandidates.

Shown are system processing hardware (102), database hardware (104), anon-select segment (112), a candidate segment (114), a scholar segment(116), a client device (120), a network (122), network interfacehardware (124), an envelope (130) representing postal delivery, ascanning and processing system (132), a workstation (140), a workstation(142) labeled “workstation n”, a correspondence printer (150), and acheck printer (152).

The system processing hardware (102) is coupled to the database hardware(104). The database hardware (104) as shown comprises the non-selectsegment (112), the candidate segment (114), and the scholar segment(116). The client device (120) is coupled to the network (122). Thenetwork (122) is coupled to the network interface hardware (124), which,in turn, is coupled to the system processing hardware (102). Theenvelope (130) representing postal delivery is coupled to the scanningsystem (132) which, in turn, is coupled to the system processinghardware (102). The workstation (140) and the workstation (142) labeled“workstation n” are both coupled to the system processing hardware(102). The correspondence printer (150) and the check printer (152) arealso both coupled to the system processing hardware (102).

The system processing hardware (102) carries out processing required forthe competition system (100). The system processing hardware (102) canbe implemented by any hardware suitable for running general purposesoftware such as, but not limited to, any commercial computing hardware.By way of example, most computing systems having one or more Intel-,AMD-, or Motorola-based processors and targeted for general home orbusiness use can be used for the system processing hardware (102).

The database hardware (104) stores information relating to candidatesrequired in carrying out the processing of competitions. The exactnature of the storage architecture is not dependant on the presentinvention and can take any form necessary to the particularimplementation. In one embodiment, as shown, the database hardware (104)is segmented to manage individuals during various stages ofconsideration in the competition system (100). In this embodiment, datais marked to segment or group the data. The data is segmented or groupedinto the categories which include applicants (the applicant category orsegment (114)), applicants who are not selected (the non-select categoryor segment (112)), and candidates selected as competition winners (thecompetition-winner category or segment (116) also called the scholarcategory or segment in one embodiment). In one variation, the databasecan be any suitable commercial database product such as, but not limitedto, products by Oracle, Sybase, or Microsoft.

In another embodiment, the database hardware (104) is physicallydifferentiated into two or more separate databases as required. In oneembodiment, the database hardware (104) includes a staging or applicantdatabase, a non-select database, a candidate or nominee database (forapplicants who have passed qualification and eligibility processes), anda competition-winner or scholar database. This embodiment is used, forexample, where existing hardware capacities mandate such an approach orin systems which have an absolute requirement of data separation such asfor security reasons. In systems requiring higher separation of data, afifth database for the storage of finalists prior to confirmation, afinalist database, can be added.

In an embodiment, applications, to be complete, comprise two or moredocuments. In a further variation, an applicant package (also called acandidate package or profile) includes an applicant form (submitted bythe applicant), also called a nominee or candidate form, a nominatorform (submitted by an individual with academic knowledge of theapplicant such as a school teacher or administrator), and a recommenderform (submitted by an individual with personal knowledge of theapplicant such as a family friend. It is noted that school officials andteachers can also be nominators). In this embodiment, the applicantsegment (114) comprises applicant-related submissions received from boththe electronic and hardcopy submissions. The non-select segment (112)comprises applicant packages for applicants who ultimately do not win acompetition, i.e., for applicants who are ultimately not selected or forapplicants having incomplete or ineligible applicant packages. Thecompetition-winner segment (116) (also called the scholar segment inacademic scholarship embodiments) comprises candidate packages relatingto those candidates who ultimately win a competition.

When application documents are received by the system, they are storedinto the database (104), associated with any other records (the databaseversions of submitted documents) relating to the same applicant, andmarked as being in the applicant segment (114). Later, the associatedrecords (i.e. the applicant packages) are submitted to qualification andeligibility procedures. Applicant packages do not pass either thequalification and eligibility procedures are marked as being in thenon-select segment (112). Later still, the applicant packages which havenot been non-selected are evaluated, scored, and one or more competitionwinners selected. In one embodiment, once an applicant is selected as acompetition finalist, he or she is referred to as a candidate and theirapplicant package is referred to as a candidate package. The candidatepackages for those candidates who ultimately win a competition aremarked as being in the competition-winner segment (116) uponcertification (verification or confirmation of data supplied in theapplicant, nominator, and recommender forms) and acceptance ofcompetition winning status (i.e. acceptance of the competition award) bythe candidate. Further to this embodiment, candidates who decline thecompetition award, or who are not certified, are designated asnon-selects and their candidate packages are marked as being in thenon-select segment (112).

In one embodiment, in order to keep track of the status of individualsduring the competition process, various lists are maintained or theability to create them at any time is preserved. These lists include anapplicant list for listing all applicants for which any document or formhas been submitted, a non-select list for listing all individuals whoultimately are not winners of the competition, a candidate list for allapplicants passing an initial qualification and eligibility process, afinalist list for listing all candidates selected as finalists, and acompetition-winner list for listing all finalists who pass aconfirmation and acceptance process. Further to this embodiment, theselists are protected and each require (possibly different) special systemprivileges to have access. In a variation of this embodiment, each listcan include various data regarding the individuals listed. Thenon-select list can include, for example, the date and stage within thecompetition at which the individual was non-selected or thecompetition-winner list can include dates at which the competitionwinner is expected to submit requalification documents.

Information relating to applicants can be submitted by any conventionalmethod such as by postal mail or electronic submission. Applicantinformation submitted by postal mail generally takes the form of a paperdocument (including predefined forms after user completion ornonstandard documents such as letters), but other manifestations, suchas, but not limited to, computer readable media (containingwordprocessing files or even image scans of paper documents) arepossible. As used herein, the term “document” refers to any submissionsof applicant information and includes, for example, paper submissions,scanned images on computer-readable media, or any other media or vehiclecontaining information to be incorporated into the competition system.

Electronic submission begins when a submitted document is transmitted bythe client device (120) via the network (122) to the competition system(100) where it is received via the network interface hardware (124). Theclient device (120) can be any user device able to initiate documentsubmissions. By way of example, a client device (120) can be, but is notlimited to, a computer, a handheld device such as a personal digitalassistant (PDA), or a computer network such as a business, college, oruniversity intranet. The network (122) can be any electronic or computernetwork or interconnection of networks such as one or more intranets orinternets. In one embodiment, the network (122) is the Internet. Thenetwork interface hardware (124) is the physical connection of thecompetition system (100) to the network (122). The network interfacehardware (124) can be any device such as, but not limited to, a modem, aT1 interface, and so forth. Electronic submissions are discussed ingreater detail later herein in reference to FIG. 5.

Documents submitted by postal mail are scanned in at the scanning system(132) and thereby incorporated into the system. In the case ofsubmissions on computer-readable media, the candidate information isread in at a computer workstation such as the workstation (140) or(142). Hardcopy submissions are discussed in greater detail later hereinin reference to FIG. 6.

In one embodiment, the overall selection process has several stagesincluding an application stage, a qualification stage, an evaluationstage, a ranking and selection stage, a finalist stage, a confirmationstage, and a tracking or monitoring stage. The application stage iswhere individual applications are received by the competition system(100) as discussed previously. The qualification stage implements basicqualification and eligibility requirements and is discussed in greaterdetail later herein in reference to FIGS. 8 and 9. The evaluation stageimplements evaluation of qualified applicants and is discussed ingreater detail later herein in reference to FIG. 10. The ranking andselection stage implements the ranking of evaluated candidates and theselection of finalists. Finalists are candidates who are selected afterthe evaluation process, but who must pass a certification process beforebecoming unqualified competition winners. The ranking and selectionprocess is discussed in greater detail later herein in reference toFIGS. 12 a-12 b.

Once finalists have been identified, the confirmation stage implementsconfirmation of the qualifications of finalists and allows for ultimateacceptance by confirmed finalists. The confirmation of finalists isdiscussed in greater detail later herein in reference to FIG. 13. Thetracking or monitoring stage implements the tracking of both competitionwinners (i.e., monitoring the progress of finalists who have acceptedtheir competition awards) and non-selected candidates (for comparisonpurposes). Competition award payment and correspondence, such as noticeletters and requests for progress reports, are generated by the paymentand correspondence generation system during the tracking and monitoringstage. The tracking of competition winners is discussed in greaterdetail later herein in reference to FIG. 14.

The workstation (140) facilitates user use of the competition system(100). The competition system can be used though a single workstationbut can also be implemented with any number of workstations necessary asindicated by the workstation (142) labeled “workstation n”. Theworkstation (140) can be any computing device able to allow display andentry of information such as, but not limited to, a personal computer, ahandheld computing device, a laptop computer, a notebook computer, aSPARC™ station, etc. The workstation (140) has at least a display, suchas a text or graphical monitor to display information and one or moreinput devices such as a keyboard and a mouse to allow entry andmanipulation of information.

The correspondence printer (150) prints correspondence and the checkprinter (152) prints checks. While two printers are shown, any suitablenumber of printers can be used such as a report printer for use by thereporting module (280) discussed in greater detail later herein inreference to FIG. 2. Alternatively, printing can be partially or totallyoutsourced.

Referring to FIG. 1 a, shown is a block diagram of an embodiment fordatabase (104) of FIG. 1.

Shown are the database (104), an applicant database (162), a scholardatabase (164), an analytical database (166), an assessment database(168), a research database (170), and an archive database (172).

In this embodiment, the applicant database (162) and the scholardatabase (164) are equivalent, respectively, to the applicant segment(114) and the competition-winner segment (116) of FIG. 1. The analyticaldatabase (166) stores information relating to analytical studies ofcandidates in the system to determine trends. The assessment database(168) stores information relating to assessments and other input fromvendor, donor, and other entities regarding the competition system(100). The research database (170) stores information relating toresearch regarding improvement of the rubric or application thereof usedby the competition system (100). Generally, the results stored in theanalytical database (166) is developed from the data stored in theresearch database (170). The archive database (172) stores any datawhich needs to be archived.

Referring to FIG. 2, shown is a block diagram of an embodiment of ahigh-level software architecture for the system of FIG. 1.

Shown are a document reception subsystem (210), a database interfacingsubsystem (220), a qualification and eligibility subsystem (230), anevaluation and selection subsystem (240), a confirmation subsystem(250), a tracking and re-qualification subsystem (260), a correspondenceand check printing interface subsystem (270), a reporting module (280),an external interface module (290), and a financial subsystem (295).

The document reception subsystem (210) is coupled to the databaseinterfacing subsystem (220), the confirmation subsystem (250), and thetracking and re-qualification subsystem (260). The qualification andeligibility subsystem (230) is coupled to both the evaluation andselection subsystem (240) and the database interfacing subsystem (220).The evaluation and selection subsystem (240) is also coupled to theconfirmation subsystem (250) and the database interfacing subsystem(220). The confirmation subsystem (250) is additionally coupled to thedatabase interfacing subsystem (220), the tracking and re-qualificationsubsystem (260), and the financial subsystem (295). The tracking andre-qualification subsystem (260) is also additionally coupled to thedatabase interfacing subsystem (220). The financial subsystem (295) iscoupled to the confirmation subsystem (250), the tracking andre-qualification subsystem (260), and the correspondence and checkprinting interface subsystem (270). The reporting module (280) and theexternal interface module (290) are coupled to each other and both arecoupled to all the other subsystems.

The document reception subsystem (210) handles the incorporation ofincoming documents into the competition system (100). While documentscan be received in a variety of ways, such as electronically or by mail,the document reception subsystem (210) incorporates the documents intothe system in a flat file format which is an electronic file containingthe data of the original document. Flat files contain their data in apredefined format understandable by the various subsystems of thecompetition system (100). In one embodiment, the flat file format is acomma delimited format such as a comma separated value (CSV) format.This stored version of a document is referred to as a record, databaserecord, or document record.

The qualification and eligibility subsystem (230) implements basicqualification and eligibility requirements. The qualification andeligibility subsystem (230), in one embodiment, has one or more filters.A filter applies one or more criteria to an applicant package anddetermines whether the applicant package meets or does not meet thecriteria. In one embodiment, applicant packages which do not meet anycriteria are disqualified and marked so as to be in the non-selectsegment (112). In other embodiments, some criteria can be discretionaryand applicants are given the chance to correct deficiencies. In oneembodiment, the filters initially determine applicant packagecompleteness (both data completeness and document record completeness).In this embodiment, once an applicant package is determined to havecompleteness, the applicant package is marked to be in the candidatesegment (114) and is now called a candidate package. After this, furtherfilters apply basic eligibility and discretionary eligibilityrequirements on the candidate packages.

The evaluation and selection subsystem (240) implements evaluation andscoring of candidate packages. The evaluation and selection subsystem(240) further compare scored candidate packages against each other anddetermines competition finalists. Competition finalists are thosecandidates which have been selected as competition winners, but arestill required to pass a confirmation and acceptance procedure. Inanother embodiment, the confirmation and acceptance procedures aredispensed with, and the candidates selected by the evaluation andselection subsystem (240) directly become competition winners. Thisembodiment is used in competitions which do not place furtherresponsibility or requirements on competition winners. By way ofexample, awards of recognition, such as a literary award, can be givenwithout requiring confirmation or acceptance.

The confirmation subsystem (250) implements procedures for confirmingfinalists as competition winners and also implements the printing ofcorrespondence alerting a finalist of his or her status and notifyingthem of the requirements and deadlines for submitting confirmationdocuments. In one embodiment, if a finalist passes the confirmation andacceptance procedures, the finalist becomes a competition winner and thecandidate package of the finalist is marked as being in theconfirmation-winner segment.

The tracking and re-qualification subsystem (260) implements theprocedures for tracking competition winners which includes the printingof correspondence notifying and alerting a competition winner todeadlines and requirements for staying qualified and the actualre-qualification determination for competition winners. Re-qualificationis imposed on competition winners for whom a continuing performancecriteria is required. By way of example, an academic scholarship mayrequire a continuing minimum level of performance as measured by gradepoint average (GPA) or a minimum level of course difficulty.

The correspondence and check printing interface subsystem (270)interfaces between other subsystems and the printing hardware forprinting correspondence, such as on the correspondence printer (150),and checks, such as on the check printer (152).

The database interfacing subsystem (220) allows the other subsystemsaccess to the database hardware (104) of the competition system (100).

The reporting module (280) generates any reports documenting processesor results which take place. These reports include, but are not limitedto, error reports such as scanning error or recognition error reports,status reports such as applicant status reports, audit reports (e.g.audit trails), possible duplicate reports, complete and incompletecandidate package reports, measurement reports, candidate detailreports, unmatched document (i.e. floater) reports, progress reports,documentation reports, and donor or other external reports.Documentation reports can include reports which document when and whatprocedures are used during competitions. Such documents can be valuablein a legal sense for sensitive competitions in which the losingcandidates may attempt to make allegations of impropriety. Donor andother external reports can be produced ad hoc to provide specific anddetailed information to the recipients. Donors, for example, may wish toknow how their donations are used and reports addressing this mayinclude amounts paid out, when, and how apportioned.

The external system interface module (290) enables the system tointerface to other external application systems such as, but not limitedto, financial systems, human resources (HR) systems, and registrationsystems.

The financial subsystem (295) implements the financial and accountingaspects of the system including keeping track of donations, awardpayments, and system costs. In an embodiment for the payment of academicscholarships, the financial subsystem (295) manages the award paymentsto scholars. In this embodiment, once an applicant is confirmed as afinalist, the financial subsystem (295) begins keeping track of thescholar. The financial subsystem (295) does this in yearly installmentsby reviewing the scholars package (i.e. the set of documents retained bythe system on the scholar) and calculates a current term award bysumming the monetary amounts of all scholarships (including 3^(rd) partyscholarships), grants, and so forth and subtracts this amount from thetotal tuition amount. The resulting amount is divided by the number ofsemesters, trimesters, etc. applicable to arrive at a base award amount,also called the current term award. Note that in this embodiment, moneyfrom programs such as loans, work study, etc. are not included in thescholarship and grant calculation as the goal of this embodiment is toallow the student to maximize study time without incurring debt forlater repayment. After arriving at the current term award, the financialsubsystem (295) calculates an adjustment to the current term award whichtakes into account the effects of short-term scholarships, differencesin tuition due to special circumstances such as study abroad or changinginstitutions, etc. to arrive at a corrected current term award. Thisamount is then certified and results in payment generated and given tothe scholar.

The financial subsystem (295) updates the previous calculations duringthe academic year at each term by reviewing the revised scholar awardpackage and calculating a supplemental award based on the review of therevised scholar award package. The revised scholar award package isgenerated by adding term-specific information to the scholar awardpackage. It is noted that the financial subsystem (295) accommodatesscholarship payments for summer terms, study abroad.

Referring to FIGS. 3 a-3 b, shown is a block diagram showing details ofthe subsystems of the software architecture of FIG. 2.

Shown in FIG. 3 a are a document reception subsystem (210), aqualification subsystem (230), a selection of competition winnerssubsystem (240), and a database interfacing subsystem (220). Each ofthese subsystems comprise various modules that are discussedindividually as follows.

The document reception subsystem (210) is shown having a networkinterface module (211), a manual input module (212) a scanner interfacemodule (213), a field extraction and OCR module (214), and an imagestorage (215). OCR is an acronym for optical character recognition.

The scanner interface module (213) is coupled to the field extractionand OCR module (214). The field extraction and OCR module (214) iscoupled to the image storage (215). The network interface module (211),manual input module (212), and the field extraction and OCR module (214)are all coupled to the duplicate check/database interface module (221).The image storage (215) is coupled to both the confirmation module (252)in FIG. 3 b (via the connector “A”), and the re-qualification module(263) in FIG. 3 b (via the connector “B”).

The document reception subsystem (210) handles the incorporation ofincoming documents into the competition system. In one embodiment, asshown, documents can be incorporated by any of three methods: electronicsubmission, hardcopy submission, and manual entry. Electronicsubmissions are received through the network interface module (211) andcan originate from any electronic source such as an internet (e.g. theInternet) or an intranet. Electronic submissions can also be acceptedfrom computers coupled directly to the competition system (and locallytransferring documents in flat file format received on magnetic disc orcompact disc, for example) such as the workstation (140) or theworkstation (142). In one embodiment, the process of electronicsubmission directly produces the flat file format of the inputted formas the user (i.e. the applicant or, in one embodiment, the recommenderor nominator) interacts with the competition system (100) in enteringdata. Electronic data entry by users is discussed in greater detaillater herein in reference to FIG. 5.

From the network interface module (211), flat files are transferred tothe duplicate check/database interface module (221). Incorporation offlat files is discussed in greater detail later herein in reference toFIG. 7.

Documents can also be submitted in paper or other hardcopy formincluding bit-image data on computer readable media such as on magneticdisc or optical disc, for example. Bit-image data refers to files not inflat format, such as bit-mapped images. Media containing bit-mappedimage data generally results from external scanning of paper documentsand storage onto computer-readable media, most likely by the candidatehimself or herself. In the embodiment as shown, the process of hardcopysubmission is received through the scanner interface module (213).Paper, or other hardcopy documents requiring scanning, are first scannedin at one or more scanner modules and incorporated onto the system via ascanner interface module (213). Pre-scanned forms submitted oncomputer-readable media are simply read into the system. Incorporationof documents via scanning is discussed in greater detail later herein inreference to FIG. 6.

Documents submitted in a non-standard form format or which otherwisefail the attempt by the field extraction and OCR module (214) to extractfield information can be incorporated into the system via rekeying atthe manual input module (212). This directly produces the correct flatfile format and this is then transferred to the duplicate check/databaseinterface module (221) where the system storage is first checked todetermine if the received flat file is already stored in the databasehardware (104).

The scanned forms of the documents are then are transferred to the fieldextraction and OCR module (214) where the information of each field isextracted and the flat file form of the document is created. Fieldextraction involves the action of extracting information frompredetermined fields and creating a “flat file” for the scanneddocument. The field extraction process requires that the documents besubmitted using a predefined form (a form having defined “fields” whichare located at predefined locations and are predetermined to holdspecified information such as for a last name, first name, etc.) whichis available upon request or from the Internet website. Use of thepredefined form is encouraged for ease of data incorporation and toensure that all required information is presented in sufficient detail.Once the flat file of a document is created by the field extraction andOCR module (214), it is transferred to the duplicate check/databaseinterface module (221).

If the flat file already exists as a record in the database hardware(104), the system does not immediately store it but determines whichversion (stored record versus received flat file) is more completelyfilled out and will retain that copy. This process is described ingreater detail later herein in reference to FIG. 7.

In one embodiment, a cutoff date is defined after which no furthersubmissions are accepted. In the case of mailed submissions, thepostmark date is the date used to determine timeliness of submission.Thus, further to this embodiment, an extension time period (such as 10days) is employed to allow timely mailed hardcopies and other mailedmedia to arrive at the competition system. Once it is determined thatall documents relating to candidate applications have been incorporatedinto the system, the qualification and eligibility subsystem (230)implements basic qualification and eligibility processes.

The qualification and eligibility subsystem (230) is shown comprising acompleteness filter module (232), a qualification filter module (233), apartner filter module (234), and an eligibility filter module (235).

The completeness filter module (232) is coupled to the qualificationfilter module (233). The qualification filter module (233) is coupled tothe partner group filter module (234). The partner group filter module(234) is coupled to the eligibility filter module (235). Thecompleteness filter module (232), the qualification filter module (233),the partner filter module (234), and the eligibility filter module (235)are all coupled to both the non-select segment interface module (222)and the candidate segment interface module (223).

The completeness filter module (232) (also called the level 1 filtermodule) analyzes applicant packages for completeness. In one embodiment,the completeness filter module (232) checks applicant packages for bothdocument completeness (i.e., that all required applicant documents arepresent) and information completeness (i.e., that all required data orinformation is present). Any applicant package which fails eitherdocument completeness or information completeness is transferred to thenon-select database and the applicant package is deleted from theapplicant list and entered in the non-select list. The application oflevel 1 filters is discussed in greater detail later herein in referenceto FIG. 8.

The qualification filter module (233) (or level 2 filter module)analyzes candidate packages to determine if the candidates meet variousminimum qualification standards. By way of example, these minimumstandards can include a minimum GPA standard.

The partner group filter module (234) (also called the level 3 filtermodule) analyzes the candidate packages to determine if the candidatesmeet various partner group-specific minimum standards. Partner groupsare subpools of candidates (i.e., less than the full pool of candidatesin the competition system) based on any one or more candidate-specificcriteria. Partner groups are created so as to provide the ability toevaluate, rank, and select winners in ways sensitive to the specificpartner group. By way of example, partner groups might be set up in anacademic scholarship setting by looking to such variables as thecandidates' geographical location, ethnic culture, poverty level, mothertongue, national origin, age, and so on. Thus, for a partner group forcandidates having a non-English mother tongue, abilities in Englishmight not be used during filtering as having little or no correlation tothe academic abilities of candidates in that partner group. Conversely,English abilities might be stressed in the filtering of partner groupsfor which English as a mother tongue was a criteria.

The eligibility filter module (235) (or level 4 filter module) imposesone or more heightened minimum response standards. In one embodiment fora competition system having multiple optional essay questions, forexample, a minimum number of essay responses are required by theeligibility filter.

It is noted that the use of more than one filter level or the use ofpartner groups is not necessary and having all filtering done by onesoftware module is possible. The separation of filters into levelsallows for easier use of criteria having different purposes. Separationof filters also allows for better separation of criteria which is usefulwhen different individuals or committees are responsible for differentfilter criteria.

The operation of the level 2-4 filters is discussed in greater detaillater herein in reference to FIG. 9.

The evaluation and selection subsystem (240) implements evaluation ofapplicant packages and selection of competition winners.

The evaluation and selection subsystem (240) is shown having a queuewith reader module (241), a reader eligibility module (242), a candidateeligibility module (243), a scoring module (244), a read verificationmodule (245), a phase 1 selection module (246), a phase 2 selection ortie-breaking module (247), and a reader monitoring module (248).

The queue with reader module (241) is coupled with the readereligibility module (242). The reader eligibility module (242) is coupledto the candidate eligibility module (243) and the queue with readermodule (241). The candidate eligibility module (243) is coupled to thescoring module (244) and the non-select segment interface module (222).The scoring module (244) is coupled to the read verification module(245) and the reader monitoring module (248). The read verificationmodule (245) is coupled to the phase 1 selection module (246) and thequeue with reader module (241). The phase 1 selection module (246) iscoupled to the phase 2 selection or tie-breaking module (247). Thereader eligibility module (242), the candidate eligibility module (243),the scoring module (244), the phase 1 selection module (246), and thephase 2 selection or tie-breaking module (247) are all coupled to thecandidate segment interface module (223).

The queue with reader module (241) queues candidate packages withevaluators or readers for later evaluation (also called reading). In oneembodiment, each applicant package is matched and queued with oneevaluator. Matching is performed by comparing information from candidatepackages with potential evaluators. In one embodiment, candidatepackages are matched to evaluators by one or more of the followinginformation: cultural background, ethnic background, age, or geographicresidence. Each evaluator thereafter should have one or more candidatepackages listed in the queue for that evaluator.

The reader eligibility module (242) requires the evaluator to pass aneligibility procedure for each candidate package prior to evaluation.This process requires the evaluator to answer one or moreevaluator/reader eligibility questions with respect to a particularcandidate package. The reader eligibility module (242) determines fromthe evaluator's responses whether the evaluator is able to evaluate thecandidate package or not. In one embodiment, for example, evaluatorquestions can be concerned with whether the evaluator attended the sameschool as the candidate or has other connections with the candidatewhich could raise questions of bias. If the evaluator is unable toevaluate the candidate, the evaluator is blocked from evaluating thecandidate and the candidate package is requeued with another evaluatorat the queue with reader module (241).

The candidate eligibility module (243) requires the evaluator todetermine whether the candidate is eligible. This determination can besimply a verification check of the same requirements enforced by thefilters in the qualification and eligibility subsystem (230) butadditional or alternative requirements can be applied by this module.

The scoring module (244) implements the actual evaluation of applicantpackages by evaluators. An embodiment for an algorithm implementingevaluation or reading of candidates is discussed in detail later hereinin reference to FIG. 11. The read verification module (245) verifiesthat the candidate package was evaluated by the required number ofevaluators. For example, in one embodiment, each candidate package mustbe evaluated by two evaluators, thus candidate packages having fewerthan two evaluations are requeued with another evaluator at the queuewith reader module (241).

The phase 1 selection module (246) handles initial selection ofcandidate finalists who, if confirmed at the confirmation subsystem(250) described hereinafter, become competition winners. The phase 2selection or tie-breaking module (247) handles the second phase or stageof candidate finalist selection. The phase 2 selection module (247)employs a finer selection paradigm than the phase 1 selection module(246) and is used to select between candidates who are very similar inoverall evaluation scores. An algorithm for the phase 1 selection module(246) is discussed in detail later herein in reference to FIG. 12 a andan algorithm for the phase 2 selection module (247) is discussed indetail later herein in reference to FIG. 12 b.

The reader monitoring module (248) implements monitoring of theevaluators during evaluations and can terminate an in-progressevaluation or nullify a completed evaluation if a problem is detected.Monitoring can take several forms. In one embodiment, evaluatorresponses are compared against corresponding reference standards. Forexample, if a competition involves evaluating candidates over severalcriteria such as writing skill and other criteria, monitoring willinvolve comparing the evaluator's responses for these criteria tostandards. The evaluator's response to writing skill, for example, canbe compared to the average score the evaluator has given to writingskill in all previous evaluations completed by that evaluator. Anothermethod is to compare the evaluator's responses to a predeterminedstandard or to the average across all evaluators. If an evaluator isfound to be erratic, corrective measures such as imposing a break,counseling, or so forth can be mandated. Additionally, completedevaluations found to be erratic can be nullified and the candidatepackage requeued with another evaluator.

The database interfacing subsystem (220) provides the necessaryinterfacing between the database hardware (104) and the rest of thesystem. The database interfacing subsystem (220) comprises a duplicatecheck/database interface module (221), a non-select segment interfacemodule (222), a candidate segment interface module (223), and acompetition-winner segment interface module (224).

The duplicate check/database interface module (221) is coupled to thedatabase hardware (104), the network interface module (211), the manualinput module (212), the field extraction and OCR module (214), and thecompleteness filtering module (232). The duplicate check/databaseinterface module (221) supplies and retrieves flat files from thedatabase hardware (104). Additionally, the duplicate check portion ofthe duplicate check/database interface module (221) determines whether aflat file to be stored already exists in the database hardware (104) asa record, and if so, determines which of the two are to be stored as themost complete version and which is to be “set aside” (i.e. stored butindicated as a rejected duplicate). The duplicate check/databaseinterface module (221) also, during the storing of flat files,determines if any other records in the database (104) relate to the sameapplicant as the file to be stored, and if so, they are associatedtogether. For example, in systems requiring an applicant package toconsist of an applicant form, a nominator form, and a recommender form,the duplicate check/database interface module (221), when storing adocument, will check if the other two forms are present for the formbeing stored, and if one or both is, the forms will be associatedtogether as an applicant package. An algorithm for implementing theduplicate check/database interface module (221) is discussed in greaterdetail in reference to FIG. 7.

The non-select segment interface module (222) is coupled to thecompleteness filter module (232), the qualification filter module (233),the partner group filter module (234), the eligibility filter module(235), and the candidate eligibility module (243). The non-selectsegment interface module (222) interfaces other software modules withthe non-select segment (112) which holds candidate packages ofcandidates who ultimately do not become competition winners.

The candidate segment interface module (223) is coupled to thecompleteness filter module (232), the qualification filter module (233),the partner filter module (234), the eligibility filter module (235),the reader eligibility module (242), the nominee eligibility module(243), the scoring module (244), the read verification module (245), thephase 1 selection module (246), and the phase 2 selection ortie-breaking module (247). The candidate segment interface module (223)interfaces other software modules to the candidate segment (114) foraccess to candidate packages which have not been marked as non-selected.

The competition-winner segment interface module (224) is coupled to thephase 1 selection module (246), the phase 2 selection module (26), theconfirmation module (252) (via the connector “C”), and there-qualification module (263) (via the connector “D”). Thecompetition-winner segment interface module (224) interfaces othersoftware modules to the competition-winner segment (116) for storage ofand access to candidate packages marked as competition winners.

Referring to FIG. 3 b, shown is part 2/2 of a block diagram in greaterdetail of the software architecture of FIG. 2.

Shown are a confirmation subsystem (250), a tracking andre-qualification subsystem and a payment and correspondence generationsubsystem (270). Each of these subsystems has various modules that arediscussed in greater detail as follows.

The confirmation subsystem (250) is shown having a confirmation documentrequest module (251) and a confirmation module (252). The confirmationdocument request module (251) is coupled to the confirmation module(252) and the correspondence printing interface module (271). Theconfirmation module (252) is coupled to the correspondence printinginterface module (271), the check printing interface module (272) andthe competition winner tracking module (261). The confirmation subsystem(250) implements confirmation of candidate finalists.

The confirmation document request module (251) determines what documentsare needed to confirm a candidate finalist and handles the generation ofcorrespondence to request these documents. The confirmation module (252)implements the actual confirmation of candidate finalists and handlesthe generation of award or rejection letters and, in the case ofwinners, initial check issuance if appropriate.

An exemplary algorithm for the operation of the confirmation subsystem(250) is discussed in greater detail later herein with respect to FIG.13.

The tracking and re-qualification module (260) is shown having acompetition winner tracking module (261), a re-qualification documentrequest module (262), and a re-qualification module (263).

The tracking and re-qualification module (260) implements progresstracking of competition winners and submits the competition winners tore-qualification to ensure that they continually meet any requirementsof the competition system (100). For example, in one embodiment forproviding academic scholarships, competition winners (scholars) havedefined academic requirements in order to continue receiving scholarshipaward payments. The competition winner tracking module (261) is coupledto the re-qualification document request module (262). There-qualification document request module (262) is coupled to there-qualification module (263) and the correspondence printing interfacemodule (271). The re-qualification module (263) is coupled to thecorrespondence printing interface module (271), the check printinginterface module (272), the competition-winner segment interface module(224), and the document receiving module (215).

The competition winner tracking module (261) determines whenre-qualification or progress documents or information is needed. There-qualification document request module (262) handles the schedulingand printing of correspondence to the necessary recipients which requestthe submission of required documents or other information so thatcompetition winner tracking and re-qualification can be carried out. There-qualification module (263) implements the actual re-qualificationevaluation of competition winners.

An exemplary algorithm for the tracking of competition winners isdiscussed in greater detail later herein in reference to FIG. 14.

The payment and correspondence generation subsystem (270) is shownhaving a correspondence printing interface (271) and a check printinginterface module (272).

The payment and correspondence generation subsystem (270) implementsgeneration of checks and correspondence. The correspondence printinginterface (271) is coupled to the confirmation document request module(250), the confirmation module (252), the re-qualification documentrequest module (262), and the re-qualification module (263). The checkprinting interface module (272) is coupled to the confirmation module(252) and the re-qualification module (263).

The correspondence printing interface (271) interfaces withwordprocessors or other end devices such as the printer (150) to printcorrespondence necessary for implementation of any other processes inthe system. The check printing interface module (272) enables theprinting of competition award checks such as on the printer (152).

The reporting module (280) is shown with multiple interconnectionsradiating outward indicating that it is coupled with substantially everyother module shown in FIGS. 3 a and 3 b. The reporting module, aspreviously discussed, collects information and generates reports. As anexample of the detail which the reporting module (280) collects, thereporting module (280) is coupled to the queue with reader module (241)from which it determines each evaluator that an applicant package isqueued to. From the reader eligibility module (242), the reportingmodule (280) documents the results of the reader evaluation, and so on.From these data, the reporting module (280) can generate reportsdocumenting exactly what occurred with a particular applicant package aswell as the performance of a particular reader, for example.

The external systems interface module (290) is likewise coupled to allmodules necessary in FIGS. 3 a and 3 b. As discussed previously herein,the external systems interface module (290) interfaces the modules andsubsystems shown in FIGS. 3 a and 3 b with external systems such asfinance systems, human resources systems, and so forth.

Referring to FIG. 3 c, shown is the block diagram of FIG. 3 a and thereporting module (280) showing interconnections of the reporting module(280). Similarly, referring to FIG. 3 d, shown is the block diagram ofFIG. 3 b showing interconnections of the reporting module (280).

Shown in FIG. 3 c are all of the modules shown in FIG. 3 a and thereporting module (280). Shown in FIG. 3 d are all of the modules of FIG.3 b. The reporting module (280) is shown in both figures connecting withall the shown modules. In operation, the reporting module (280) recordsthe events of each module for documentation purposes. The reportingmodule (280) is able to synthesize a great variety of reports from therecorded information including, but not limited to document submissionerror reports, scanning error reports, possible duplicate reports, listsof all applicants for a specified competition, lists of all floaterdocuments, status reports on a specified applicant package or aspecified set of applicant packages (such as by an entered list or aspecified partner group or a group defined by a specified piece or rangeof information), evaluation summary reports by applicant, partner group,evaluator, scholar financial reports, scholar progress reports, dataanalysis reports, trend analysis reports, and so forth. The reportingmodule also stores and can reference in reports the documenting ofdecisions by competition system personnel. Such documentation providesproof, if later required, of the decisions made by system administratorsand personnel. Decisions which may be documented in this way includedecisions regarding duplicate document and fraud situations, the orderof evaluation variable application for tie-breaking, what constitutes acomplete document, and so forth. It is noted that one of ordinary skillin the art, in implementing a competition system (100) for a definedpurpose and a defined reporting capability, would know what modules tocouple to the reporting module and what data the reporting module wouldaccumulate and store. Additionally, one of ordinary skill in the art,given a defined purpose and a defined reporting capability, would knowwhich modules the external systems interface module (295) would need tobe coupled to and what transfer protocols would need to be implemented.

Referring to FIG. 4, shown is a general flowchart of the steps traversedin operation of the systems of FIGS. 1 and 2.

The competition system (100) receives (402) application documents whichcan originate from applicants themselves and/or other individuals. Inone embodiment, as discussed previously, are applications and documentsin support thereof from nominators or recommenders. These submissionscan be via electronic means such as, but not limited to, electronicsubmissions via the network (122) such as via the Internet, or viahardcopy submissions such as paper copies submitted by postal mail(130). In the case of hardcopy submissions using a preformatted formdesigned for the system of the present invention, the hardcopy isscanned using a scanning system (132). During the scanning process, formfields are identified and the information contained therein is extractedand read. In one embodiment, the scanning is done by optical means usinga commercially available scanner although any other method, such as bymechanical means, can be implemented.

In one embodiment, regardless of method of submission, forms anddocuments must be converted to a flat file format discussed previouslyin reference to FIG. 2. In submissions via the network (122), the flatfile is created directly from the inputted information. In submissionsincorporated via the scanning system (132), the scanning system producesthe flat file from the data retrieved during field extraction andoptical character recognition (OCR). Similarly, flat files are generatedfrom electronic submissions on computer-readable media or manual entryvia a workstation such as workstation (140).

Once the flat file form of received documents are present, the systemstores (404) the flat files in the database hardware (104) as recordsand they are marked as being in the staging segment (110). At the timeof storing, the system groups (404) or associates the flat file with anyother records from the same applicant. It is at this stage that thevarious records relating to the same applicant are associated to formapplicant packages. While it is possible that duplicates of one or moreof these forms can be submitted to the system, only one of each formtype is included in an applicant package. Any duplicate forms areretained but identified as duplicates (the process of identifying theduplicate forms and excluding them from the candidate packages is termed“setting aside”).

In one embodiment, the association of records is done by applicantsocial security number. In this embodiment, the applicant record (theapplication submitted by the applicant his- or herself) is the key file(or key record). Once the applicant form is stored as a record, theapplicant identified in the record is added to an applicant list. If anyother required forms identifying that applicant are later submitted,such as a nominator form or a recommender form, they are associated withthe corresponding applicant record or key file. Required forms such asnominator forms or recommender forms, which are submitted without acorresponding key form (the applicant form) being present are termed“floaters” indicating they do not have a key file to be associated to.

After the cutoff date (406), further electronic submissions via thenetwork interface module (211) are prevented. The cutoff date is chosenso as to provide enough time to carry out the rest of the competitionprocessing prior to any necessary deadlines. In one embodiment, ifhardcopy forms submitted by mail or other means are allowed in acompetition, the postmark dates of mailed submissions are used todetermine whether a submission was timely. When using postmarked datesto determine cutoff timeliness, extra time is allowed beyond the cutoffdate (such as 10 days) to allow for the physical delivery of the mailedsubmissions to the competition processing site and the necessarysubsequent incorporation of the hardcopy data into the competitionsystem. All applicants identified in any form which was received by thecutoff date and the postal mail extension time period allowed for areidentified and recognized within the system and will therefore beconsidered during the competition.

At this point, the competition system checks (408) the completeness ofthe applicant packages. This determination is also referred to as level1 filtering. In one embodiment, the filtering is done in two or morestages or levels which are described in greater detail in reference toFIG. 8 and FIG. 9. In one embodiment, the completeness of applicantpackages is checked in two ways. One determination is whether anapplicant package has document completeness. As discussed previously inreference to one embodiment, a complete applicant package is comprisedof an applicant or nominee form, a nominator form, and a recommenderform. Any package determined not to contain all required initial formsis determined to be incomplete. A second determination is whether eachdocument in a candidate package contains all required data orinformation. As each document required in a complete candidate packageis different from all others, each document generally has differentrequired data or information standards. Applicant packages which arelacking one or more required documents or which contain documentsmissing one or more required datum or pieces of information aredetermined to be incomplete. These applicant packages are redesignatedas non-selects and are removed from the applicant list, added to thenon-select list, and, along with any duplicate forms, are marked to bein the non-select segment (112). All complete applicant packages whichmeet the basic eligibility requirements are marked to be in thecandidate database (114). Thereafter, these applicant packages arereferred to as candidate packages. Candidate package are then filtered(412) with one or more higher level filters.

After candidate packages pass the filtering process, they are read(414). Reading involves evaluation of the candidates with respect to therubric of the competition system. The term rubric refers to thevariables and the scoring process which are used to score or select thewinning candidate. The variables can be objective or quantitative (suchas grade point average or GPA) or subjective or qualitative (such as theresults of essay evaluation). The variables can also include bothcognitive variables (GPA, essay writing, etc.) or non-cognitivevariables (leadership experience, presence of a support person, etc.).In one embodiment, evaluation or reading is done by humans who aretrained to evaluate the candidate packages, although automated readingimplemented by software means or a combination of automated processesand human evaluation can be implemented. In one embodiment, humanreaders evaluate the candidate packages against a rubric composed in thegreater part by non-cognitive variables such as, but not limited to:positive self-concept, realistic self-appraisal,understanding/navigation of social systems, preference for long-termgoals over short-term needs, availability of a support personage,leadership experience, community service, and non-scholastic knowledgeconcentration (i.e. appreciable knowledge acquisition about a field notcovered in past academic environments).

Evaluation or reading is generally done multiple times for eachcandidate package for reasons of reducing the effects of spurious readdeviations. In one embodiment, the reading process is completed twoindependent times for each candidate package (i.e., the candidatepackages are each evaluated by two different readers). In otherembodiments, however, multiple partial reads are used which allowsexpert readers to specialize on a subset of evaluation variables. Instill other embodiments, only one read or more than two reads can beused per candidate depending on the repeatable accuracy of readers, thelimitations of the competition finances, or the fairness safety factordesired. The process of evaluation or reading is described in greaterdetail later herein in reference to FIG. 10.

In order to ensure the accuracy and repeatability of human (andautomated) readers, reader normalization (or reader norming for short)is implemented in one embodiment. Reader normalization refers to thoseprocesses implemented to ensure readers operate within a predeterminedvariance standard during their evaluation. Reader normalization is usedduring training and continued during the actual reading of candidates.As a component of reader norming during actual evaluations, readermonitoring is implemented which allows for detection of errantevaluations or biases. Reader norming is discussed in greater detaillater herein in reference to FIG. 11.

Once the candidate packages have been read, the candidate finalists areselected (416). Selection generally is implemented by selecting thecandidates with the highest read scores. The selection process can becomplicated and involved when the number of competition winners ispredetermined and the number of read variables is relatively low becausegenerally there will be a read score for which some candidates will beselected, but due to the limited number of total competition winners,others at the same score will not be able to be selected. The selectionprocess is described in greater detail later herein in reference toFIGS. 12 a and 12 b.

Once the candidate finalists are determined (416), the candidatefinalists are put through a confirmation process (418) which involvesverifying statements made in the candidate's application (the nomineeform in an embodiment) and any other required documentation (such as thenominator and recommender forms, in an embodiment) and also verifyingthat the candidate meets any eligibility requirements the data for whichwere not known at the time of application. The process of confirmationis aided by the reception (420) of required confirmation documents. Byway of example, in an academic scholarship competition it may be thatthe candidate does not know which college or university he or she willattend at the time of application, but this information must be knownand supplied prior to the academic scholarship grant. The process ofconfirmation is discussed in greater detail later herein with respect toFIG. 13.

After confirmation, competition winners are tracked (422) to monitortheir progress and, optionally, to determine whether or not thecompetition winner is maintaining any post-competition requirements. Inan academic scholarship system, for example, post-competitionrequirements may relate to academic performance, and if met, the systemresponds by continuing payment of the scholarship award. Thepost-competition tracking system is discussed in greater detail laterherein in reference with FIG. 14.

Referring to FIG. 5, shown is a flowchart of an exemplary algorithm(500) for the steps traversed in the operation of electronic documentsubmission of FIG. 4.

As one method of receiving information from potential candidates, awebsite is maintained which allows access and online informationsubmission by individuals. When an individual accesses (502) thewebsite, for security reasons, the website requires (504) identificationof the accessing individual by a logon protocol. Additionally, in oneembodiment, security is implemented during the submission process. Byway of example, digital signatures are used and retained by the systemfor verification. Alternatively, the system can require or allow the useof personal identification smart cards to verify identity.

Once the individual is successfully logged on, the system will providethe individual with options which include online form completion andsubmission. The individual then indicates his or her role (506) (e.g.,in one embodiment as applicant, nominator, or recommender) and, in thecase of individuals providing support documents, the identity of theapplicant (504) for which they wish to submit a form. After selectingonline form creation, the system which first determines (508) whetherthat form was already submitted (i.e. whether that individual alreadysubmitted a form for the candidate identified). Duplicate submissionsare sometimes requested for various reasons such as the individualwishes to change some information, the individual forgets the priorsubmission, and so on. In order to prevent creation of duplicatedocuments, the system prevents (510) the individual from continuing ifthe form requested has already been submitted to the system.

In one embodiment, duplicate forms are detected using the candidate'ssocial security number only. In another embodiment, potential duplicateapplications are detected by the use of the last name, the first letterof the first name, the date of birth, and the state of birth and/orresidence. In this embodiment, potential duplicates identified by thesecriteria are included in a duplicate report and system administratorswill be able to access a special interface set up to allow simultaneousdisplay of data from two applications so that the two potentiallyduplicated forms can be viewed together. Duplicates can arise which havealternate candidate social security numbers. This can occur in the caseof a duplicate form being input with an erroneous candidate socialsecurity numbers or intentionally changed social security number (i.e.,fraud).

Detection of both innocent and intentional duplication of forms isnecessary for proper competition administration. Cases of apparentduplicates with different social security numbers do often occur and acommon situation is the case of twins. By way of example, it is notedthat in some Asian cultures, twins may receive identical ornearly-identical names and thus may appear to be a highly likely case ofduplication when in actuality it is not. In one embodiment, the systemadministrator reviews all possible duplicate situations prior to theinvolved documents continuing in the competition process. To facilitatethis, the system generates a possible duplicate report such as shown inFIGS. 22-23.

If the form requested was not previously completed, the individual ispresented (530) with the appropriate form for online completion. In theembodiment shown, the individual is interacting directly with thesystem, so the system must wait (530) for each field submission (usuallyby pressing of the enter key or the tab key). In another embodiment,forms can be completed by use of software such as a downloaded programor browser plugin and the results submitted at completion.

Next, the system determines (512) whether the submit button (indicatingthe individual has completed the form and wishes to submit the form forstorage). If the individual's action was not submission but informationentry in a field, the system determines (514) whether the entered datameets basic validity criteria for that field and presents (516) anyerror message to the individual. By way of example, if the fieldrequires the birthdate of the candidate, the system may impose a generalform requirement that the date be entered in the common mm/dd/yy form(i.e. two digit month numeral followed by two digit day numeral followedby a two digit year numeral). The system could also provide a presumedvalidity requirement by presuming invalid any birthday which is in thefuture, was in the past 10 years, or was prior to 100 years previously(under the hypothesis that no candidate would be younger than 10 yearsold or greater than 100 years old). If an error message is presented,the cursor resets (518) in the field where the invalid data was enteredand the system again waits (532) for information entry and thereaftercontinues as discussed previously.

If a field submission is determined (514) valid, the systemautomatically positions (520) the cursor in the next field andthereafter waits (532) for further information entry and thereaftercontinues as described previously.

When the individual has completed the form and selects the submitbutton, the system determines (522) whether the form is completelyfilled out. If not, the system allows (524) the saving of incompleteforms for later continuance after which the online form entry session isfinished for now as shown by done indicator (526). If the form isdetermined (522) to be complete, the form is stored in the databasehardware (104) as a record and marked (528) to be in the staging segment(110) and the form entry session is finished as shown by done indicator(526).

Referring to FIG. 6 shown is a flowchart of an exemplary algorithm (600)of the steps traversed in the operation of hardcopy submissions in thesystems of FIGS. 1-2.

Another way besides electronic submission for forms to be submitted byindividuals is by paper or hardcopy submission such as by postal mail0.Other forms of submission could also be used which include, but are notlimited to, facsimile submission, compact disc or other opticalrecording device submission, electronic file submission other thanonline form completion such as by a portable document format (PDF) file,submission on magnetic media such as tape or disc (e.g., floppy disc orharddisk), and mechanical submission such as by punched card, sheet, ortape.

As the competition system (100) receives (602) submissions, they arecollated (604) by form type (e.g. by applicant, nominator, andrecommender forms as discussed previously for one embodiment). Atperiodic intervals, or as convenient, the received forms are scanned(606) in batches. After scanning, the resultant images are submitted(610) to a field extraction and optical character recognition (OCR)process to extract the data in the images. The system then determines(612) whether the field extraction and OCR process was successful, andif it was, the algorithm (600) then creates (614) an export file andthen is done for the data extraction of that image as indicated by thedone indicator (616). After this, the algorithm (600) generally selects(602) another form for processing and continues as before until allscanned documents are likewise submitted to the field extraction and OCRprocess. If the field extraction and OCR process is determined (612) tohave not been successful, forms can be manually entered (618). Thealgorithm (600) then creates (614) an export file and then is done forthe data extraction of that image as indicated by the done indicator(616).

Generally, errors sometimes result in interpretation of fieldinformation. By way of example, an error could be as simple anoccurrence as a comma having been entered when a “/” was required (suchas in a date field). A scan report is generated which summarizes theresults of the scan process of each batch and if errors resulted, theseare indicated by document in the scan report.

Referring to FIG. 7, shown is a flowchart of an exemplary algorithm(700) of the steps traversed in the storing and grouping of recordsshown in block (404) of FIG. 4.

The algorithm (700) stores flat files into the database hardware (104)as records. These flat files are received through any of the methodsdiscussed previously in reference to FIGS. 5-6 (such as electronicallyand via postal mail with subsequent scanning or manual entry). Thealgorithm (700) begins by determining (702) the form type (i.e. in oneembodiment, whether the flat file represents a applicant form, nominatorform, or recommender form) and applicant identity from the flat file. Inone embodiment this is easily done as forms must be submitted usingpredefined forms having specified fields or identifying information ormarkings which identify the nature of the form. Next, the algorithm(700) determines (703) whether any records in the database (104) forapplicants with a different social security number have the same formtype and contain data which substantially matches the flat file to bestored. If any records are found substantially matching the file to bestored, this is a potential fraud situation and an entry is made (704)in the duplicate list.

Next, the algorithm (700) determines (705) if a record exists in thedatabase (104) which is a match for the form type and the sameapplicant. If not, the flat file is stored (706) in the databasehardware (104) as a record. If the form is an applicant form (708), itis indexed (710) in the applicant list or key list. The applicant formor key list is The algorithm (700) then checks a floater list for anyfloaters corresponding to the stored applicant record and if there areany, they are associated (712) with the just-stored applicant record.The floater list is a list of all forms (such as the nominator andrecommender forms in the prior embodiment) for which a correspondingapplicant record has not been stored in the database hardware (104). Thealgorithm (700) is then done for this flat file as indicated by the doneindicator (714).

If the flat file is not an applicant form (708), a check (716) is madeif the corresponding applicant (or key) form has already been indexed.This is done in one embodiment by use of the candidate's social securitynumber (SSN), which is also required on the nominator and recommenderforms in identifying the applicant for which the form is submitted. If acorresponding applicant form has not been received by the competitionsystem (100), the file is indexed (720) and flagged as a floater in thefloater list. After indexing of the file in the floater list, thealgorithm (700) is done with this form as shown by the done indicator(714).

If a corresponding applicant form is found (716) to have been indexed bythe system, the stored record is associated (718) with the correspondingapplicant or key form. The algorithm (700) is then done with this formas shown by the done indicator (714).

If the flat file is determined (705) to already be in the databasehardware (104), this means that the flat file is considered a possibleduplicate and is indexed in a duplicate list. The duplicate list is areport which keeps track of all duplicates. The algorithm (700) nextcompares the stored record with the received flat file to determine(722) whether the received flat file is more complete than the recordalready stored in the database hardware (104). If it is, the recordalready stored in the database hardware (104) is flagged as “set aside”and the received file is stored and is the record which is used by thecompetition system (100) during further considerations.

It is noted that as discussed previously, by consideration of theduplicate list, system administrators or other officials can verifywhether the correct action was taken.

If the received file is determined (722) to be less complete than therecord stored already, the received form is stored (724) in the databasehardware (104) and flagged as “set aside”, but is still associated (726)with the other documents of the same applicant. The algorithm (700) isthen done with this form as shown by the done indicator (714).

Referring to FIG. 7 a, shown is a flowchart of an exemplary algorithmfor duplicate detection review in one embodiment.

In one embodiment, system administrators or other authorized personnel,must review all possible duplicate detections prior to the qualificationand eligibility phases. A reviewer accesses the possible duplicate listto determine the possible duplicate situations to review. Afterselection (752) of a possible duplicate situation to review, thereviewer notes (754) the nature of the situation. If the situation isone of two or more documents identifying the same applicant socialsecurity number (SSN), the reviewer reviews the involved documents todetermine which document is more completely filled out. In oneembodiment, the duplicate detection/database interface module (291)makes an initial decision as to which document is more complete. In thisembodiment, if the reviewer determines (756) that the duplicatedetection/database interface module (291) made the right decision,documents (758) that decision and is done for that situation. Otherwise,if the reviewer determines (756) that the decision of duplicatedetection/database interface module (291) was in error, the reviewercorrects the decision and documents (758) the decision and action and isdone for that situation.

If the situation is determined (754) to be a situation which does notinvolve identical social security numbers, the reviewer notes (762)whether it is a situation of possible fraud. Possible fraud situationsresult from two or more documents bearing demographic and otherinformation which appears to refer to an identical individual but whichidentifies different social security numbers. If the situation is one ofpossible fraud, the reviewer reviews (764) the involved documents andmakes a determination. Possible determinations include (1) that theinvolved documents relate to different individuals (i.e. no fraud), (2)that the documents refer to the same individual (i.e. actual fraud), (3)that the documents refer to the same individual but that one or moresocial security numbers were improperly entered, and so forth. After thedetermination and resultant action, the reviewer documents (766) thedetermination and action and is thereafter done (760) for the situation.

If the reviewer determines (762) that the situation is something otherthan simple duplicate detection or possible fraud, the reviewer reviews(768) the situation, makes a determination, implements thedetermination, and documents (720) the determination and implementation.Thereafter, the reviewer is done (760) with the situation.

Referring to FIG. 8, shown is a flowchart of an exemplary algorithm(800) of the steps traversed in the determination of candidate packagecompleteness in FIG. 4.

Once the cutoff date (406) and any extended time for postal delivery hasbeen reached, applicant packages are analyzed for completeness. In oneembodiment, applicant packages are analyzed for both documentcompleteness and each document is analyzed for data completeness. Thealgorithm (800) begins by selecting (802) an applicant package foranalysis. The algorithm (800) determines (804) first whether theapplication package has data completeness, and if so, the algorithm(800) selects a form from the package. Applicant packages, to havedocument completeness, must be made up of a minimum required set ofdocument records. For example, in one embodiment, a candidate packagerequires an applicant form, a nominator form, and a recommender form tobe complete. The algorithm (800) then determines (808) whether the formhas data completeness. For an applicant package to have datacompleteness, each document or record in the applicant package must meeta minimum data completeness standard (such as all required fields havingnon-null entries or all required data present). Note that because eachdocument in an applicant package is different, each may have differentminimum data completeness requirements. If the record selected does havedata completeness, the algorithm (800) determines (810) whether any morerecords in the applicant package need to be checked for datacompleteness, and if so, the algorithm (800) selects (806) anotherrecord and continues as described previously. If there are no furtherrecords to be analyzed (810), the algorithm (800) is done for thispackage as indicated by the done indicator (812).

If a record is determined (808) not to have data completeness, it isflagged (814) as incomplete and the applicant package is marked (816) tobe in the non-select segment (110). Thereafter, the algorithm (800) isdone for this package as indicated by the done indicator (812).

If the applicant package is determined (804) to be lacking in one ormore records, the applicant package is flagged (818) as incomplete anddesignated (816) as a non-select by being marked as in the non-selectsegment (110) and being listed in the non-select list. Thereafter, thealgorithm (800) is done for this package as indicated by the doneindicator (812).

Referring to FIG. 9, shown is a flowchart of an exemplary algorithm(900) of the steps traversed in the level 2-3 filtering of candidatepackages shown in FIG. 4.

After the candidate packages have passed the level 1 filters of datacompleteness and form completeness, they are filtered by one or morehigher level filters. In one embodiment, a candidate package is filtered(902) by level 2 filters which ensure candidates meet certain qualifyingcharacteristics. Further to this embodiment these qualifyingcharacteristics include a minimum GPA score or completion of a generalequivalency degree (GED) and minimum financial requirements (such as amaximum family income to be eligible for receiving an academicscholarship). The algorithm (900) then determines (904) whether thelevel 2 filter qualifications are met and if they are not, the candidatepackage is marked (906) to be in the non-select segment (112) and thecandidate is listed in the non-select list.

Otherwise, if the algorithm (900) determines (904) that the level 2filter qualifications are met, further filters can be applied to thecandidate package. In one embodiment, a candidate package is filtered(908) by level 3 filters which ensure that the candidate meets certainheightened response requirements. Further to this embodiment, the level3 filters are applied specific to the candidate's subgroup or partnergroup. In this embodiment, once candidates pass the level 1 and level 2filters, they are grouped together into two or more subgroups or partnergroups. In one embodiment, the grouping is done by the creation ofpartner lists. The grouping can be done based on any one or more piecesof data. By way of example, grouping could be done in an academicscholarship setting by looking to such variables as the candidates'geographical location, ethnic culture, poverty level, mother tongue, andso on. The level 3 filters are then be applied by subgroup/partner groupand ensure minimum qualifications designed in light of the specificcharacteristics of the particular subgroup.

In one embodiment, group or partner administrators are able to apply thelevel 3 filters more than one time in order to reduce the candidate poolfor that candidate group or partner group to predetermined levelsselected as optimum prior to going into the read or evaluation process.Criteria applied during level 3 filtering includes, in one embodiment, aminimum number of academic awards, a minimum number of public awards, aminimum number of honors, a minimum number of leadership roles, aminimum number of student excel bubbles completed, a minimum class rigoraverage, a minimum amount of community service, a maximum amount ofpersonal circumstance exceptions, and a minimum number of paid hours ofemployment.

In another embodiment, the level 3 filters, in order to effectivelyreduce the pool of candidates to a desirable number, are predetermined.In this embodiment, prior to the start of the competition process, apredetermined set of discretionary criteria and an ordering of theirapplication is determined. Additionally, a desired pool size or a targetrange within which the pool must number is selected. Thereafter, in theactual filtering process, the discretionary criteria is applied inincrements and the number of applicants remaining in the pool isascertained after each increment, with the filtering stopping when thetarget pool size is reached. By way of example, the first 10discretionary criteria in one embodiment are, in a competition foremployment at a law firm, a law degree from a 2^(nd) tier school orhigher, a grade point average of 3.2 or higher in a law degree,completion of one or more engineering degrees, a grade point average of3.2 or higher for one engineering degree, the attendance of a top 20engineering school for one engineering degree, a law degree from a1^(st) tier school, a grade point average of 3.5 or higher in a lawdegree, completion of two or more engineering degrees, a grade pointaverage of 3.5 or higher in an engineering degree, and the attendance ofa top 10 engineering school for one engineering degree. Thus, thesecriteria would be applied in order until the pool of applicants wasreduced to the desired level.

In another embodiment, the level 3 filters, in order to effectivelyreduce the pool of candidates to a desirable number, are determined as afunction of the number of applicants in the applicant pool. In thisembodiment, the discretionary criteria is predetermined but instead oftheir application being dependent on a required outcome, theirapplication is dependent on the initial size of the applicant pool. Inthis variation, the criteria itself can be dependent on the size of thepool of candidates. For example, again in a system for the selection ofa law firm associate, one criteria for a pool of applicants below 50 isthat the applicants must have a 3.2 grade point average or higher in alaw degree and for a pool of 100 or more candidates, the criteria is aminimum of a 3.5 grade point average. Alternatively, the criteria couldbe that the applicants must have a law school grade point average whichexceeds 4−(1/x) where x is the number of applicants in the pool. Aftereither of the block (910) or the block (906), the algorithm (900) isfinished as shown by done indicator (912).

If the candidate package passes (910) the level 3 filters, the candidatepackage has passed the filtering phase of the competition as shown bythe done indicator (912). If the candidate package does not pass (910)the level 3 filtering, it is stored (906) in the non-select database(112) and the candidate is listed in the non-select list.

Additional or fewer levels of filtering can be applied within the scopeof the present embodiment. In one variation, an additional level 4 setof filters is applied which ensure that the candidate has responded to aminimum number of essay questions in his or her applicant form. Thesystem applies the level 4 filters in a manner similar to that asdescribed for the level 2 and level 3 filters.

Referring to FIG. 10, shown is a flowchart of an exemplary algorithm(1000) of the steps traversed in operation of the reading process inFIG. 4.

The system begins by queuing (1002) a applicant package with anevaluator (also referred to as a reader) who has not previouslyevaluated that applicant package or been disqualified from evaluating orreading that applicant package. The evaluator then begins by responding(1004) to a set of reader qualification questions which document whetherthe evaluator has any connections or conflicts with the applicant whichwould raise questions of bias. By way of example, in one embodiment,reader qualification questions include whether the evaluator served asnominator or recommender for the applicant and whether the evaluator hasany connection with the applicant such as having attended the same highschool or college as the applicant or that the evaluator has anypersonal reason not to read the applicant. In this example, a yes to anyreader eligibility question disqualifies the reader from reading theapplicant.

The system then checks (1006) whether the reader is qualified to readthe applicant, and if not, the reader is disqualified (1008) fromreading the applicant and the system again queues (1002) the applicantpackage with another available reader.

After verification (1006) of reader eligibility to evaluate or read theapplicant package, the evaluator verifies (1010) whether the applicantis eligible based on the requirements of the competition system. Tofacilitate this, the system presents the necessary information extractedfrom the applicant package on the reader's display along with astatement of applicant eligibility requirements. The system thendetermines (1012) from the reader's verification whether the applicantis eligible and if the applicant is not eligible, the system marks theapplicant package to be in the non-select segment (112) and indexes theapplicant in the non-select list.

If the applicant is determined (1012) to be ineligible for reading, thealgorithm (1000) marks (1014) the applicant package to be in thenon-select segment (112) and lists (1014) the applicant in thenon-select list. In this case, the algorithm (1000) is thereafterfinished as shown by the done indicator (1020).

If the applicant is (1012) eligible, the reader then reads (1016) theapplicant package. Reading is facilitated by the side-by-side display ofthe read variable scoring area and an information display. FIGS. 20-21show views of an exemplary display for reading. In the informationdisplay, the system can display applicant data segments which aresections of data extracted from one or more forms in the applicantpackage or the system can display complete forms. The system also blocksinformation which is of a personal nature such as the applicant's socialsecurity number (SSN), email address, mailing address, alienregistration number, date of alien registration number issuance, andfamily information such as gross annual income, family size, and numberof family members currently in college. When the reader clicks orselects a variable, the system will pull up the relevant data segmentsor applicant package forms and displays them in the information area foruse by the reader.

The read variables are determined by the administrators of thecompetition system and the scores achieved by applicants on the readvariables are what determines whether they are ultimately successful inthe competition. By way of example, the read variables for oneembodiment in the academic scholarship arena include the non-cognitivevariables 1) positive self-concept, 2) realistic self-appraisal, 3)understanding/navigation of a social system, 4) preference of long-termgoals over short-term goals, 5) availability of a strong support person,6) leadership experience, community service, 7) interest/knowledge in anon-school field, 8) community service, as well as the cognitive-relatedvariables 9) curriculum rigor, 10) academic achievement, 11) essayresponse.

In one embodiment, the scoring during evaluation is by a numerical scaleas follows: 1 indicates a negative score, 2 indicates a neutral score, 3indicates a positive score, and 4 indicates an excellent score. In somevariable cases, less than the total number of scores will be available.By way of example, in the case of the non-cognitive variable presence ofa strong support person, only the two scores of 2 or 3 are allowed.

After the reader has completed and submitted the reading of theapplicant package, the system determines (1018) whether the applicantpackage has been read by the required number of different readers. Useof multiple readings has many benefits including, but not limited to,ensuring greater fairness and providing compensation for the event ofone reader providing a randomly “unfair”, inequitable, or inaccurateread which is not detected by the system. In one embodiment, twoindependent reads are required but more reads can be used within thescope of the invention. If the required number of readers has not readthe applicant package, the system queues (1002) the applicant packagewith another available reader and continues as before. Otherwise, thereading phase for the applicant package is complete as shown by the doneindicator (1020).

Referring to FIG. 11, shown is a flowchart of the steps traversed in theoperation of reader selection and training for participation in thealgorithm (1000) of FIG. 10.

First, the number of readers is determined (1102). In one embodiment,this is done manually by the system administrator based on the number ofcandidates anticipated and either (a) the number of reads per readerdesired or (b) the length of the reading phase considered in conjunctionwith the average number of reads per reader per day. Once the number ofdesired readers is determined, reader applications are collected (1104)and evaluated (1106). In one embodiment, readers are filtered to meetminimum criteria related to the nature of the competition to beconducted. By way of example, for a competition system for militaryofficer selection, readers would likely be required to have minimumqualifications which include military experience or familiarity, officerselection experience, and so forth. The administrators of the systemthen select (1106) the readers.

The readers are then trained (1108) with the goals and procedure of thecompetition system. Following the initial orientation, the readers aretrained (1110) about the read variables and the scoring rubric. Intraining the readers on the scoring process, the readers are firsttrained (1112) by paper and pencil scoring exercises which facilitates apersonal review procedure whereby trainers may personally discuss thescoring rubric with the reader trainee. Once the readers havedemonstrated a basic competence during the paper exercises, the readersare then trained (1114) by computer-based exercises which substantiallymimic the actual reading displays and use actual prior candidatepackages (without personal identifying information). As with the paperexercises, reader trainers are available to reinforce the scoring rubricby discussions of the methodology and balancing of factors which go intoscoring the read variables. In closing the electronic reading exercises,an evaluation is conducted (1116) to ensure that the readers are readingat the level mandated by the competition system requirements. After thereaders pass the evaluation, the reader training phase is finished asshown by the done indicator (1118).

Referring to FIGS. 12 a-12 b, shown is a flowchart of an exemplaryalgorithm (1200) of the steps traversed in operation of finalistselection in FIG. 4.

Referring to FIG. 12 a, shown is part 1/2 of the algorithm (1200).Finalist selection occurs (1202) after the reading is finished. Thenumber of competition winners to be chosen is read (1204) (oralternatively input by a system operator) and the read variable scoresfor each candidate are summed (1206) to produce read variable sums. Byway of example, in one embodiment, 11 read variables are scored eachscored by two evaluators. Thus, for each candidate, there are 22 totalread variable scores which are then summed together to provide the readvariable sum for that candidate. After summing the read scores, softwarevariables to be used in later processing are initialized (1208). Thecandidates are then grouped (1210) by read variable sums. By way ofexample, if 25 candidates out of a pool of 400 candidates achieve a readvariable sum of 80, then these 25 candidates form one group and theother 375 candidates form one or more other groups as dictated by theirread variable sums.

Once the candidates are clustered into groups by read variable sums, thegroups are ranked (1212) from highest read score sum to next lowest readscore sum and so on until all groups are ranked. At this point, theactual selection process begins by first selecting (1214) the highestranking group as the active group. Next, the algorithm (1200) determines(1216) whether the total number of candidates already selected (zero atthis point) plus the number of candidates in the active group exceedsthe total number of competition winners to be selected. If the totalnumber of candidates already selected (again, zero at this point) plusthe number of candidates in the active group is less than the totalnumber of competition winners to be selected, all candidates in theactive group are selected (1218). Selection entails the candidates beingdesignated as finalists by a entry being made in the finalist list andtheir candidate packages being flagged as a finalist. The algorithm(1200) then returns to block (1214) and selects (1214) the next highestranking group as the active group and continues as described beforeexcept that now, the variable describing the total number of candidatesalready selected is no longer zero. The loop consisting of the stepsdescribed, (1214) to (1218) inclusive, continues until the firstoccurrence where the total number of candidates already selected plusthe number of candidates in the active group is determined to exceed(1216) the total number of competition winners to be selected. At thispoint, the algorithm (1200) leaves the loop, steps (1214) to (1218), andcontinues (via the connector “C” (1220)) to the second level or phase 2of selection which begins at the step (1222) shown in FIG. 12 b.

Referring to FIG. 12 b, shown is part 2/2 of the algorithm (1200). Thispart of the algorithm (1200) is an exemplary embodiment fortie-breaking. Tie-breaking refers to the method of selecting a subset ofan active group of candidates in order to end up with the same number ofselected candidates (i.e. finalists) as the total number of competitionwinners to be selected.

The embodiment shown in FIG. 12 b uses 11 read variables during thetie-breaking process. However, part 2/2 of the algorithm (1200) as shownin FIG. 12 b can be used with any number of read variables. The block(1222) is a loop control which limits the number of iterations to amaximum equal to the number of read variables available (11 readvariables in the shown embodiment). First, the algorithm (1200)determines (1224) whether the number of selected candidates (finalists)exactly equals the total number of competition winners to be selected,and if it does, the selection process is finished as indicated by doneindicator (1226). Otherwise, the algorithm (1200) selects (1228) thehighest priority read variable which was not already selected and reads(1230) the weight to be applied to that read variable.

Read variable priority is a ranking or scaling of the various readvariables for use during tie-breaking. The read variable priorities are,in an embodiment and for reasons of fairness, determined prior to thereading process. In one embodiment, read variable priorities aredetermined by partner group and not globally over the entire pool ofcandidates. In this embodiment, the partner groups are free to determinethe read variable priorities in ways which makes the most sense relativeto the characteristics of the candidates of that partner group. The readvariable having the highest priority will be used first in anytie-breaking which needs to be done. Then, if the total number ofselected candidates is not yet reached, the next highest variable isused and so on until the total number of allotted awards are matched toselected candidates.

Once the highest priority read variable is determined, it is weighted(1232) by a predetermined amount. Further to the last embodiment, thepredetermined amount, like the priority ordering of the read variables,is determined before the reading process. Weighting refers tomultiplying the selected read variable by the predetermined weight foreach candidate in the active group. By way of example, for a competitionsystem applying two reads over 11 variables for each candidate, thereare two read variables for each candidate which are weighted and thepredetermined weight to be applied during first round tie-breaking is 2.After weighting the read variables, a new read variable sum is computed(1234). Next, the candidates (of only the active group) are firstclustered (1236) into subgroups or cells such that (a) each subgroup hascandidates all having the same new read variable sum (which incorporatesthe weighted variables) and (b) all candidates having the same new readvariable sum are in the same subgroup. Next, the subgroups are ranked(1238) by from highest weighted read variable sum progressively down tothe lowest weighted read variable sum.

On the first pass through the tie-breaking loop (block (1222) to block(1248) inclusive), only one read variable weight is applied. On eachsuccessive iteration through the tie-breaking loop another read variableis weighted. Read score sums in a given iteration through thetie-breaking loop incorporate the weighted read scores of all previousiterations (e.g. once a read variable is weighted, the weighted variableis used for the rest of the tie-breaking rather than reverting to theoriginal non-weighted score).

The tie-breaking selection process now continues similarly to theselection process as described in reference to FIG. 12 a. The highestsubgroup is designated (1240) as the active subgroup. Next, thealgorithm (1200) determines (1242) whether the total number of selectedcandidates plus the candidates in the active subgroup exceeds the totalnumber of competition winners to be selected. If not, the candidates inthe active subgroup are selected (1244) and the algorithm (1200)continues back and selects (1240) the next highest ranked subgroup asthe active subgroup and continues as before. The act of selecting of acandidate means that the candidate has been converted to finaliststatus. At this point, all that is done is that the candidate is indexedinto a selected list. The loop consisting of block (1240) to block(1244) is iterated until the total number of selected candidates plusthe candidates in the active subgroup is determined to exceed (1242) thetotal number of competition winners to be selected. When this occurs,the highest ranking subgroup having candidates who have not beenselected is designated (1246) the active group. Next, the algorithm(1200) checks (1248) whether the total number of competition winners tobe selected equals the number of selected candidates and if it has,algorithm (1200) has finished the selection process (as shown by block950). Otherwise, the algorithm (1200) continues back and determines(1222) the next highest priority read variable and continues as before.

Generally, this embodiment of tie-breaking is very effective innarrowing out of a group of candidates the necessary subgroup to fillout the total number of candidates to be selected. The only complicationis when two or more candidates receive exactly the same score on eachread variable and also just happen to fall within the read scorehierarchy such that only a subset of them can be chosen to fill out theallotment of awards. In this case, successive weighting and ranking willnot be able to differentiate between them so as to fill out the awardallotments. This situation can be handled by having backup variableswhich normally are not used for scoring and ranking which can be used.These backup variables could also be scored by the readers, oralternatively, they could be taken from candidate information such asGPA, normalized GPA, etc.

Once all of the finalists have been selected, the system implements theactual conversion of the candidates to finalists by designating allcandidates as a finalists in the their respective candidate packages,adding the candidates to the finalist list, and deleting the candidatesfrom the nominee list.

It is noted that in one embodiment, candidate finalists are subject toconfirmation and if confirmed, are required to accept prior to beingconfirmed as a competition winner. In this embodiment, some candidatefinalists will not be confirmed and others who are confirmed may notaccept. Thus, in order for the predetermined number of competitionwinners to be met, the system must select new competition finalists foreach competition finalist who is ultimately not confirmed or who doesnot accept even when confirmed. In one embodiment, the system selectsadditional finalists by continuing from where the phase 1 and phase 2selection systems left off. The selection of additional finalists can bedone periodically, such as once a week, or can be done more frequentlysuch as once a day or even each time notification of non-acceptance ornon-confirmance is received.

Referring to FIGS. 12 c-12 e, shown is an example of an application oftie-breaking. Referring to FIG. 12 c, shown is an example of thepriority ordering of evaluation variables for two partner groups in oneembodiment.

As an example of tie-breaking, one embodiment concerns the selection ofindividuals for employment. In this embodiment, the variables over whichthe candidate packages are evaluated are curriculum rigor, overallacademic achievement, positive self-esteem, realistic self appraisal,understanding and navigation of social and organizational systems,preference for long-term goals over immediate gratification, leadershipexperience, community service, self-motivated acquisition of knowledgeor skills, persuasiveness, language structure and expression ability.These variables are chosen to maximize the predictive ability of thecompetition system and it is noted that in competitions employed forother purposes or with different goals, these variables may be chosen todifferently.

In this embodiment, applicants are divided into partner groups whichinclude partner groups for engineering and business management. Withinthe partner group for engineering the evaluation variables are orderedfor priority as: overall academic achievement, curriculum rigor,language structure and expression ability, preference for long-termgoals over immediate gratification, self-motivated acquisition ofknowledge or skills, understanding and navigation of social andorganizational systems, leadership experience, persuasiveness, positiveself-esteem, realistic self appraisal, and community service. Incomparison, within the partner group for business management theevaluation variables are ordered for priority as: understanding andnavigation of social and organizational systems, leadership experience,positive self-esteem, language structure and expression ability,persuasiveness, preference for long-term goals over immediategratification, self-motivated acquisition of knowledge or skills,overall academic achievement, curriculum rigor, realistic selfappraisal, and community service. It is noted these orderings perpartner group reflect the goals of the particular competition and wouldlikely be chosen differently for different competition objectives.

Referring to FIG. 12 d, shown are an example of the evaluation resultsfor two candidates in an embodiment.

As an example, two candidates are in competition for an engineeringposition. As part of the qualification and eligibility process, each wasdetermined to have the necessary engineering degrees and so on. Duringthe reading process, candidate # 1 received for reader #1: 4, 4, 2, 3,3, 2, 4, 2, 3 and, for reader #2: 3, 4, 2, 2, 4, 2, 3, 4, 4, 3, 2.Candidate #2 received for reader #1: 4, 4, 3, 3, 3, 2, 4, 4, 3, 2, 2 andfor reader #2: 2, 4, 4, 4, 3, 2, 4, 2, 2, 3, 2.

Referring to FIG. 12 e, shown is an example of tie-breaking between thetwo candidates of FIG. 12 d according to the evaluation variablepriorities of FIG. 12 c.

Thus, in phase 1 of the selection process, both candidates have anoverall score of 66 and so a decision of which to choose cannot be made.In a situation where only one of these candidates is to be selected,resort is made to the phase 2 or tie-breaking selection process whichuses evaluation variable priorities. For the engineering partner group,the first priority variable is overall academic achievement (the secondvariable in the global listing of variables). In the example, bothcandidates have the values of 4, 4 in their two readings. If theweighting is 2, for example, these scores become 8 and 8 for bothcandidates and their overall scores become 74 for both. The secondpriority variable for the engineering groups is curriculum rigor (thefirst variable in the global evaluation variable listing). Candidate #1received scores of 3, 3 for this variable whereas candidate #2 received4, 2. Applying a weighting of 2 again, these scores become 6, 6 and 8,4, respectively and both candidates receive an overall score of 82. Thethird priority variable for the engineering partner group is languagestructure and expression ability (the last variable in the globalevaluation variable order) for which candidate #1 received 3, 2 andcandidate #2 received 2, 2. Thus, applying a weighting of 2 for thisvariable results in scores of 6, 4 and 4, 4, respectively. Thus,candidate #1 now has a total score of 87 and candidate #2 now has ascore of 86 and the tie has been broken in favor of candidate #1.

Referring to FIG. 13 shows a flowchart of an exemplary algorithm (1300)of the steps traversed in operation of competition finalist confirmationshown in FIG. 4.

Competition finalist confirmation refers in part to the verification ofvarious data presented on candidate package forms and in part to verifythe eligibility of the candidate relating to any new information such asmay now be available concerning the tuition needs of the candidatefinalist. In one embodiment, once candidates have passed the reading andselection phases, they are transferred to a scholar tracking module.Entry into the scholar tracking module allows for individual access byadministrators as various confirmation documents arrive and areincorporated into the system and also allows manual update ofinformation and candidate status. The scholar tracking system allows forthe administrator to search by social security number (SSN), partialSSN, specific surname, and generic surname. The term generic surnamerefers to the entry of an initial set of letters followed by a wildcard.The result of a generic surname search is a list of all names matchingthe query. By way of example, the query “Mus*” would result in a listcontaining Musam, Muse, and Musni provided those names were in thedatabase queried.

The algorithm (1300) causes (1302) correspondence to be sent to thecandidate finalists which announces their finalist status and whichrequests all necessary confirmation documents. In one embodiment, therequired confirmation documents include a complete high schooltranscript (covering at least the last 3 years attended) or a generalequivalency degree (GED) as appropriate, any tribal documents if thefinalist is Native American (note that some Native American tribes haveonly one document to indicate tribal membership whereas others may havemultiple documents which together indicate tribal membership), a UnitedNegro College Fund (UNCF) Scholars information form (discussed in detailin the next paragraph), a copy of the finalist's college/universityadmissions letter indicating admittance, a letter/document for eachscholarship (outside of this competition) which the finalist has beenawarded indicating the amount of award, and any documents for each loanthe finalist has indicating the amount and terms of the loan. In avariation of this embodiment, confirmation documents additionallyinclude documents attesting to citizenship or primary residency,affidavits and documents regarding community service, documentsregarding employment, documents regarding honors and awards, andaffidavits and documents regarding leadership experience.

The UNCF Scholars information form is a form developed to standardizeinstitution information disclosure relating to financial grants tostudents. It has been determined through actual experience that thelevels of detail provided by colleges and universities in response toinquiries concerning financial grants to students can vary wildly frominstitution to institution. For example, some universities and collegesdo not report 3^(rd) party scholarships in student financial reports. Inresponse to this, a standardized form was developed which specifies whatinformation needs to be disclosed. In one embodiment, the UNCF Scholarsinformation form requires disclosure of all scholarships, grants, loans,and other financial assistance to the student, both from the academicinstitution as well as 3^(rd) parties. This level of detail is needed,in this embodiment, as the goal of the competition system in which it isused is to (1) complete the financial needs of competition winners byproviding any required academic tuition not met by financial assistancealready in place and additionally to repay in full all loans (so thestudent will not have that obligation later).

Once confirmation documents are received, they are incorporated into thesystem by the document receiving subsystem embodiments of which aresimilar to the procedure described in reference to FIGS. 4 and 5 exceptthat documents are stored (1304) in the candidate database and linkedimmediately with the rest of the corresponding candidate package.

In the actual confirmation process, a candidate package is selected(1306). The algorithm (1300) then determines (1308) whether all requireddocuments are present. If all the required confirmation documents arepresent, the candidate confirmation documents are evaluated (1310). Inone embodiment, the evaluation of candidate confirmation data is done bya system administrator. The algorithm (1300) then determines (1312)whether the candidate has been confirmed, and if so, the candidate isconverted (1314) to scholar status. Conversion to competition winnerstatus includes the marking of the candidate package as being in thecompetition-winner segment (116) and the addition of the candidate to acompetition winner list. The competition winner list is the master listof all competition winners for use by the system. From this point on,the candidate is referred to as a competition winner and his or herpackage is referred to as a competition winner package.

After conversion (1314) of a candidate to a competition winner, thealgorithm (1300) determines (1316) whether any more candidateconfirmations are outstanding, and if so, the algorithm (1300) selects(1308) another candidate package and continues as before. If thealgorithm (1300) determines (1316) that no candidate packages areoutstanding, the confirmation process is finished as shown by the doneindicator (1316).

If the algorithm (1300) determines (1308) that one or more requiredconfirmation documents are missing, the candidate is converted (1320) toa non-select by marking the candidate package as being non-select andlisting the candidate in the non-select list. Thereafter, the algorithm(1300) determines (1316) whether any candidate packages still need to beconfirmed, and continues as described previously.

If the algorithm (1300) determines (1312) that the candidate is notconfirmed, the candidate is converted (1320) to a non-select asdescribed previously and the algorithm (1300) thereafter determines(1316) whether any candidate packages still need to be confirmed, andcontinues as described previously.

Referring to FIG. 14 shows a flowchart of an exemplary algorithm (1400)of the steps traversed in operation of competition winner tracking andpayment and tracking in FIG. 3.

While the algorithm (1400) is optimized for the administration ofacademic scholarships, it is noted that with minor adaptations, thisalgorithm can monitor the continued performance of military or civilofficers, employee performance, and so on.

Competition tracking refers to the monitoring of scholar progress,updating of competition winner files to add academic performance andother information, and re-qualification that the competition winnermeets the requirements of the competition system. As such, thecompetition winner tracking system must periodically update thecompetition winner files. To do this, the algorithm (1400), afterdetermining that it is time to re-qualify one or more competitionwinners (such as just before an academic year in an academic competitionsystem), sends (1402) correspondence requesting any needed information.In one embodiment relating to academic scholarships, academictranscripts are requested once a year at the completion of the year ofstudy. Further to this embodiment, prior to each further scholarshippayment, the scholar is asked whether he or she wishes to continue inthe scholarship.

In one embodiment, the system maintains a scholar tracking action listfor scholars in order to keep track of when re-qualification isnecessary. The system determines, by accessing the scholar trackingaction list, when an action, such as requesting re-confirmationdocuments is required. In another embodiment, this information is keptseparately for each scholar.

All re-qualification documents received in support of the scholar arestored (1404) into the competition-winner segment (116) and associatedwith the competition winner package. As with initial confirmationdocuments, documents requested in response to updating a competitionwinner's package can be received electronically or by hardcopy and areincorporated into the system by the same system and process as describedin reference to FIGS. 4 and 5, with the difference that the documentsare stored in the scholar database and associated with the scholarpackage.

Next, a first competition winner package is selected (1406). Next, thealgorithm (1400) determines (1408) whether the re-qualificationdocuments are all present and if so, the algorithm (1400) allowsevaluation (1410) of the re-qualification materials to determine if thescholar re-qualifies. Both the checks if all required documents arepresent and the evaluation of the competition winner's re-qualificationcan be implemented by automation, by human reviewers, or a combinationof both. In one embodiment relating to the field of academic scholarshipcompetitions, both the checking if all required documents are presentand the evaluation of the scholar's re-qualification are done by humanreviewers. Further to this embodiment, the algorithm (1400) allows thehuman reviewers to quickly search in scholar packages for specificdocuments by keyword, document type, document receipt date, documentoriginator, and so forth. Further, the algorithm (1400) allows forsimultaneous display of portions of, or the complete contents of, anyre-qualification materials (or indeed any form in the scholar package)with the re-qualification rubric and/or decision input form.

After evaluation (1410), the algorithm (1400) determines (1412) whetherthe competition winner has re-qualified and if so, the new amount of thescholarship award is calculated (1414). After the scholarship awardamount has been determined (1414), a check is printed and sent (1416) tothe competition winner's educational facility.

Next, the algorithm (1400) determines (1418) if all competition winnershave been passed through the re-qualification process, and if so, there-qualification process is finished (as shown by done indicator (1420).Otherwise, the algorithm (1400) selects (1406) another scholar packagewhich has not been re-qualified and continues as described before.

If the algorithm (1400) determines (1408) that not all requiredre-qualification documents are present, the algorithm (1400) alerts andfacilitates (1422) competition winner reviewers to follow up with thecompetition winner. If the re-qualification materials are competed(1424), the algorithm (1400) evaluates (1410) the scholar'sre-qualification materials and continues as discussed previously. If thealgorithm (1400) determines (1424) that the re-qualification materialsis not complete, the algorithm (1400) determines (1426) whether thecompetition winner has at least one semester of sabbatical left underthe terms of the scholarship. If the competition winner does have atleast one semester of sabbatical left, the competition winner isdesignated (1428) as being on sabbatical and the competition winner'ssabbatical allotment is decremented (1428) by one (1). Thereafter, thealgorithm (1400) determines (1418) if all competition winner packageshave been reviewed and continues as previously discussed. If thecompetition winner is determined (1426) to have no sabbaticals left, thecompetition winner's scholarship is terminated (1430) and thecompetition winner's package is updated (1430) to reflect this status.After terminating (1430) the scholar's scholarship, the algorithm (1400)determines (1418) whether any scholars have not been processed throughthe re-qualification process and continues as previously described.

If the competition winner is determined (1412) to not be re-qualified,the algorithm (1400) determines (1426) whether the competition winnerhas any sabbaticals left and continues as previously discussed.

Referring to FIG. 14 a, shown is a flowchart of the steps traversed inan exemplary method (1450) for the tracking of competition winners andnon-selected applicants.

In one embodiment for the distribution of academic scholarships, thetracking and re-qualification subsystem (260) implements tracking andinformation accumulation on both scholars and non-selected applicants.The algorithm (1450) operates by the administration of surveys and otherinformation retrieval activities based on the time of year. Thealgorithm (1450) first checks (1452) if it is the start of the academicyear, and it is, the algorithm (1450) updates scholar data throughrequest (1454) of the re-qualification materials as discussed inreference to FIG. 14. As discussed in reference to FIG. 14, once there-qualification materials are received, they are incorporated (1456)into the system. If it is not (1452) the start of the academic year, thealgorithm (1450) checks (1458) if it is the time for the orientation ofnew scholars and if it is, new scholar surveys are administered (1460)to the new scholars. Thereafter, this information is incorporated (1456)into the system. If it is not (1458) orientation time, the algorithm(1450) checks (1462) if it is time for the bi-annual surveys and if itis, career development surveys (1464) and academic support surveys(1466) are administered to the active scholars. If it is not (1462) timefor the bi-annual active scholar surveys, the algorithm (1450) checks(1468) if it is graduation time for any active scholars and if it is,graduate surveys are administered (1470) to the graduating scholars.Thereafter, this information is incorporated (1456) into the system. Ifit is not (1470) a graduation time, the algorithm (1450) waits againuntil the first of the academic year (1452) and continues as previouslydescribed.

Additionally, information supplied by active scholars and scholar alumnimay be received (1472) at any time. This can occur by mailed in materialsubmissions, facsimile transmissions, website submissions, and so forth.As with the previously discussed survey information, this information isincorporated (1456) into the system.

Information is also received (1474) from third parties regardingnon-selected applicants. This received information is very important forverifying and improving the selection rubric and criteria employed bythe system. Non-selected applicant data is extracted from third partieswhich get this information during transactions with the non-selectedapplicants during situations such as, but not limited to, applicationsto other scholarships, credit cards, colleges or universities, etc. aswell as from targeted promotions designed to elicit trackinginformation.

Improvement of the system rubric and evaluation criteria can beeffectuated by tracking the performance of both competition winners andnon-selected applicants and comparing the later performance of bothtypes of individuals with their predicted performance as derived fromthe system rubric through the evaluation variables. To do this, eachcompetition winner's tracked performance is evaluated and scored overone or more performance variables. In one embodiment, these variablesinclude annual salary, gaps between employment periods, number of peoplemanaged, number of positive appearances on the news or on television,job satisfaction, community leadership, positive contributions tosociety, and so forth. Analysis is conducted of these performancevariable scores in reference to the individual's original evaluationvariable scores to determine the accuracy of each evaluation variable inpredicting the eventual resultant performance of the individual. Thisanalysis is conducted for all competition winners and non-selectedapplicants. The accuracy of predicting eventual resultant performance ofeach performance variable can thus be determined for each evaluationvariable. Additionally, this analysis can be conducted separately forany subgroup of competition winners and for a corresponding subgroup ofnon-selected applicants to determine the significance of each evaluationvariable in correctly differentiating individuals who went on to performsignificantly from those who did not.

Referring to FIG. 15, shown is an exemplary display presented to anevaluator. An evaluator is given the opportunity to either get anapplicant for evaluation or to quit.

Referring to FIGS. 16-17, shown are two views of an exemplary displayfor use during determination of evaluator eligibility. Shown areselected information from an applicant package. In this example, shownare the candidate's gender, birthdate, dependency status,race/ethnicity, languages spoken, summary of high school attendance,general equivalency degree (GED) status, and future goals. Additionallyshown is evaluator eligibility question guidelines and three evaluatoreligibility questions for the evaluator to answer. Note that in otherembodiments, different questions and different numbers of totalquestions are used. Determination of evaluator eligibility andsubsequent redirection of applicant packages to another evaluator whenthe evaluator is determined ineligible to evaluate the applicant packagehelps ensure an unbiased evaluation. Additionally, evaluator eligibilityis documented in the system for later use, if needed, to counter anyreceived allegations of impropriety.

Referring to FIGS. 18-19, shown are two views of an exemplary displayfor use during determination of candidate eligibility. Shown isinformation extracted from a candidate package and four questionsregarding eligibility of the candidate package to be evaluated. In thisexample, the questions regarding the candidate package eligibility arewhether the candidate is eligible based on high school graduation or GEDcompletion, high school grade point average (GPA), not having attendedcollege or university, and sufficient information for evaluation. Notethat in other embodiments, different questions and different numbers oftotal questions are used.

Referring to FIGS. 20-21, shown are exemplary displays for use duringevaluation of candidate packages. Shown in FIG. 20 is the typicalsplit-screen display used in one embodiment. Shown are two windows orareas. One area (the data area) contains information from thenominator's form for the candidate. The other area (the control andresponse area) contains various controls or hyperlinks which control theinformation displayed in the first area. In the shown embodiment, thecontrols allow the evaluator to control the data area to show applicantform data, nominator form data, or recommender form data. By selecting,such as by use of a mouse, an evaluator directs the system to displayinformation from the selected record in the data area. The control andresponse area also contains 11 evaluation variables requiring evaluatorresponses. In this embodiment, shown are the variables 1) curriculumrigor, 2) overall academic achievement, 3) positive self-concept, 4)realistic self-appraisal, 5) understanding/navigation of a socialsystem, 6) preference of long-term goals over short-term goals, 7)leadership experience, community service, 9) self-directed acquisitionof knowledge in a non-school field, 8) community service, 10)availability of a strong support person, 11) structure and use oflanguage in essays. By selecting a particular one of the evaluationvariables, the evaluator directs the data area to scroll to or extractinformation from the displayed record which is relevant to the selectedvariable. For example, as shown the data area of FIG. 20 showsinformation relevant to curriculum rigor. The data area shows thisinformation whenever the evaluator indicates, by selecting thecurriculum rigor variable, that the evaluator wishes to evaluate thecurriculum rigor variable.

FIG. 20 a shows the screen shot of FIG. 20 with additional controlwindows. The window below and to the left of the two evaluation windowsas shown in FIG. 20 is a control window allowing the evaluator to accessthe stored image of a document from the applicant package underevaluation. This allows the evaluator to view the original documentimage which is displayed in the data area. This capability is also usedby authorized personnel when deciding situations of possible fraud. Thewindow shown to the right of the two evaluation windows as shown in FIG.20 is a control window allowing the evaluator to find a document view.As shown, a document can be found by searching such as by socialsecurity number and form type or content.

Referring to FIGS. 22-25, shown are exemplary displays of candidateprogress detail reports. Shown in FIG. 22 is an applicant packageprogress report detailing applicant packages grouped bycomplete/incomplete and sorted by social security number (SSN). Shown inFIG. 23 is an applicant package progress report grouped bycomplete/incomplete and sorted by applicant name. Shown in FIG. 24 is anapplicant package progress report grouped by complete/incomplete, byethnicity, and sorted by social security number (SSN). Shown in FIG. 25is an applicant package progress report grouped by complete/incomplete,by ethnicity, and sorted by applicant name. In one embodiment, access tothese reports is restricted such that only system administrators areable to bring up such reports.

Referring to FIGS. 26-28, shown are exemplary displays of completedapplicant package reports. FIG. 26 shows a report listing all completedapplicant packages sorted by applicant social security number. FIG. 27shows a report listing all completed applicant packages sorted byapplicant name. FIG. 28 shows a report listing all completed applicantpackages sorted by submit date of the applicant (nominee) form.

Referring to FIGS. 29-30, shown are exemplary displays of possibleduplicate document reports. FIG. 29 shows a listing of possibleduplicate forms listed by applicant social security number. FIG. 30shows a report of possible duplicate forms listed by applicant name.

Referring to FIGS. 31-34, shown are exemplary display of candidatedetail listing reports. Shown are candidate detail data. FIG. 31 showsapplicant detail data sorted by social security number. FIG. 32 showscandidate detail data sorted by applicant name. FIG. 33 shows applicantdetail data ordered by one partner group sorted by social securitynumber. FIG. 34 shows candidate detail data ordered by partner group andsecondly by applicant name.

Referring to FIG. 35, shown is an exemplary report of a nominatorfloater with possible applicant or key form (nominee form) ordered byapplicant last name and secondly by social security number. This reportlists all nominator floaters and likely corresponding key forms (nomineeforms), if the system discovers any.

Referring to FIGS. 36-37, shown are exemplary display of nominatorfloater detail reports. FIG. 36 shows a report listing nominator formfloaters ordered by applicant social security number. FIG. 37 shows areport listing nominator form floaters ordered by applicant last nameand secondly by social security number.

Referring to FIGS. 38-40, shown are exemplary display of nominatordetail reports. These reports list nominator detail data. FIG. 38 showsnominator detail data ordered by applicant social security number. FIG.39 shows nominator detail data ordered by applicant name. FIG. 40 showsnominator detail data ordered by nominator name.

Referring to FIG. 41, shown is an exemplary report of recommenderfloaters with possible key form (nominee form) linkages ordered byapplicant last name and secondly by social security number. This reportlists all recommender floaters and likely corresponding key forms(applicant forms), if the system discovers any.

Referring to FIGS. 42-43, shown are exemplary display of recommenderfloater detail reports. FIG. 42 shows a report listing recommender formfloaters ordered by applicant social security number. FIG. 43 shows areport listing recommender form floaters ordered by applicant last nameand secondly by social security number.

Referring to FIGS. 44-46, shown are exemplary display of recommenderdetail reports. These reports list recommender detail data. FIG. 44shows recommender detail data ordered by applicant social securitynumber. FIG. 45 shows recommender detail data ordered by applicant name.FIG. 46 shows recommender detail data ordered by recommender name.

While the invention herein disclosed has been described by means ofspecific embodiments and applications thereof, numerous modificationsand variations could be made thereto by those skilled in the art withoutdeparting from the scope of the invention set forth in the claims.

The invention claimed is:
 1. A system for evaluating and selectingcompetition winners from a pool of candidates in a competitioncomprising: a database for storing candidate documents; systemprocessing hardware configured to operate as: a grouping subsystemcoupled to said database for grouping said candidate documents intocandidate data sets; a qualification subsystem coupled to said databasefor determining said candidate data sets to be qualified ordisqualified; a pool reduction subsystem coupled to said database fordetermining whether said candidate data sets having been qualified meeta discretionary eligibility standard, said discretionary eligibilitystandard varying as a function of the number of said candidate data setshaving been qualified; an evaluation subsystem coupled to saidqualification subsystem and said database for evaluating each of saidcandidate data sets having met said discretionary eligibility standardand scoring one or more evaluation variables in response to saidevaluation; and a selection subsystem coupled to said evaluationsubsystem and database for selecting one or more competition finalistsresponsive to said evaluation and scoring of said candidate data setshaving met said discretionary eligibility standard.
 2. A system as inclaim 1 further comprising: a document reception subsystem comprising: anetwork interface module coupled to said database for receivingelectronic submissions, said database storing said electronicsubmissions as candidate documents, a scanner interface module forinterfacing said system to a scanner and receiving scans of submittedhardcopy documents, and a data extraction module coupled to said scannerinterface module and said database for extracting data from said scans,said database storing said extracted data as candidate documents.
 3. Asystem as in claim 2, further comprising: a duplicate checking modulecoupled to said network interface module, said data extraction moduleand said database for determining whether said candidate documents existin said database, said candidate documents having been determined not toexist in said database being stored in said database.
 4. A system as inclaim 1 wherein said qualification subsystem comprises: a first filtermodule coupled to said database for determining whether said candidatedata sets meet a data completeness standard.
 5. A system as in claim 4wherein said data completeness standard requires said candidate datasets to include a minimum set of document types.
 6. A system as in claim4 wherein said evaluation subsystem comprises: an evaluation workstationcoupled to said database for facilitating evaluation of said candidatedata sets having met said discretionary eligibility standard, saidevaluation workstation comprising: a display for displaying informationextracted from said candidate data sets, and an input device forentering information; a matching module coupled to said database forselecting a possible evaluator to evaluate one of said candidate datasets having met said discretionary eligibility standard; an evaluatoreligibility module coupled to said database for determining whether saidevaluator is permitted to evaluate said one of said candidate data setshaving met said discretionary eligibility standard; and an evaluationmodule coupled to said evaluator eligibility module, said database, andsaid evaluation workstation for providing on said display informationextracted from said one of said candidate data sets having met saiddiscretionary eligibility standard, providing one or more evaluationvariables for scoring, and receiving a score for each of said one ormore evaluation variables from said input device.
 7. A system as inclaim 1 wherein said selection system comprises: a phase 1 selectionmodule coupled to said evaluation subsystem for selecting a first set ofcandidate finalists responsive to total scores generated responsive tocorresponding said received scores, said first set of candidatefinalists not exceeding a predetermined total number; said phase 1selection module further outputting when said first set of candidatedata sets is less than said predetermined number a sub-pool of saidcandidate data sets all having received equal corresponding said totalscores; and a phase 2 selection module coupled to said phase 1 selectionmodule and said database for selecting a second set of candidatefinalists, said second set of candidate finalists being a subset of saidsub-pool, said first set of candidate finalists plus said second set ofsaid candidate finalists equal in number to said predetermined totalnumber.
 8. A system as in claim 7, said phase 1 selection modulecomprising: a scoring module coupled to said evaluation subsystem andsaid database for generating said total scores; a grouping modulecoupled to said scoring module and said database for grouping saidcandidate data sets into score tiers responsive to corresponding saidtotal scores, each of said score tiers comprising one or more of saidcandidate data sets having equal said total scores; a selection modulecoupled to said ranking module, said ranking workstation, and saiddatabase for selecting a first set of candidate finalists responsive tosaid ranking, said first set of candidate finalists equal to or lessthan said predetermined total number; and an output module foroutputting said sub-pool, said sub-pool of candidate profiles comprisingone of said score tiers, said one of said score tiers having the highesttotal score of those of said score tiers containing candidate profilesnot selected for said first set of candidate profiles.
 9. A system as inclaim 7, said phase 2 selection module comprising: a receiving submodulecoupled to said for receiving said sub-pool of said candidate data setsfrom said phase 1 selection module; an evaluation response selectionsubmodule for determining one of said received scores to be weighted anda weight; a weighting submodule coupled to said selection module andsaid evaluation response selection submodule for weighting said one ofsaid received scores of said candidate data sets in said sub-pool; ascoring submodule coupled to said weighting submodule for generatingweighted scores for said candidate data sets in said sub-pool; agrouping submodule coupled to said scoring submodule for grouping saidcandidate data sets in said sub-pool into sub-tiers responsive tocorresponding weighted scores; a selection submodule for selectingadditional candidate finalists responsive to said weighted evaluationtier ranking; and a stop submodule for determining when a correct numberof candidate finalists have been selected and outputting said additionalcandidate finalists.
 10. A system as in claim 1 wherein said selectionsystem comprises: a scoring module for summing evaluation scores foreach of said candidate data sets having been evaluated; a score groupingmodule for grouping said candidate data sets having been evaluated intoscore tiers each comprising substantially similar corresponding ones ofevaluation score sums; a phase 1 selection module for selecting all saidcandidate data sets of the score tiers beginning with the score tierhaving the highest median evaluation score sum and proceeding to scoretiers of successively lower median evaluation score sums such that thenumber of candidate documents selected is maximized without exceeding apredetermined number, said phase 1 selection module further outputtingan active group, said active group comprising the candidate data sets ofthe score tier having the highest median evaluation score sum containingno candidates having been selected: and a phase 2 selection module forselecting a subset of said active group.
 11. A system as in claim 10wherein said phase 2 selection system comprises: a weighting module forweighting said evaluation scores for each of said candidate data sets insaid active group; a summing module for summing said weighted evaluationscores for each of said candidate data sets in said active group; ascore grouping module for grouping said candidate data sets in saidactive group into phase 2 score tiers by substantially similarcorresponding said evaluation score sums; a tie-breaking selectionmodule for selecting all candidate documents within the phase 2 scoretiers beginning with the phase 2 score tier having the highest medianevaluation score sum and proceeding to phase 2 score tiers ofsuccessively lower median evaluation score sums such that the totalnumber of selected said candidate data sets is maximized withoutexceeding said predetermined number; and a designating module fordesignating the candidate data sets of the phase 2 score tier having thehighest median evaluation score sum containing no candidates having beenselected as the active group.
 12. A system as in claim 1 furthercomprising: a monitoring subsystem coupled to said evaluation subsystemfor monitoring evaluations of qualified said candidate data sets, saidmonitoring subsystem allowing interaction with an ongoing evaluation,said monitoring subsystem allowing nullification of an evaluation whensaid evaluation is determined to be inconsistent with the requirementsof said competition.
 13. A system as in claim 1 further comprising: aconfirmation subsystem coupled to said selection subsystem and saiddatabase for determining the confirmation status of said one or morecompetition finalists and determining one or more competition winnersresponsive to said confirmation status determination.
 14. A system as inclaim 13 wherein said confirmation subsystem comprises: a workstationcomprising: a display for displaying candidate packages andcorresponding confirmation documents, and an input device for enteringinformation; a confirmation document request module coupled to saiddatabase for requesting confirmation documents; a confirmation modulecoupled to said workstation and said database for providing saidcandidate packages and said corresponding confirmation documents to saidworkstation for display and for receiving confirmation responses; and aconversion module coupled to said workstation and said database formarking ones of said candidate data sets as non-selected responsive to acorresponding confirmation response indicating nonconfirmation and formarking ones of candidate data sets as competition winners responsive toa corresponding confirmation response indicating confirmation.
 15. Asystem as in claim 1 further comprising: a competition winner trackingsubsystem coupled to said database for tracking said competition winnerscomprising: a competition winner tracking module coupled to saiddatabase for determining when update documents are required for acompetition winner, an update document request module coupled to saidcompetition-winner module for coordinating update document requestcorrespondence, and an update module coupled to said database forupdating confirmation winner packages with said required updatedocuments.
 16. A system as in claim 15, wherein said competition winnertracking subsystem further comprises: a workstation comprising: adisplay for displaying information from one of said candidate data setsand corresponding confirmation documents, and an input device forentering information; and a reconfirmation module coupled to workstationand said database for displaying the candidate data sets of saidcompetition winners with corresponding said required documents andreceiving reconfirmation responses, the status of the candidate datasets of said competition winners being updated in response to saidreconfirmation responses.